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ABSTRACT 

This  thesis  presents  an  evaluation  of  three  relatively  simple  reli- 
ability growth  models  for  which  accuracy,  precision,  and  robustness  per- 
formance were  examined  over  a  wide  variety  of  true  underlying  reliability 
growth  patterns.   A  continuous  cumulative  failure  rate  model,  a  continu- 
ous instantaneous  failure  rate  model,  and  a  discrete  reliability  model, 
each  of  which  employ  ordinary  regression  methods,  were  evaluated  using 
standard  computer  Monte  Carlo  simulation  techniques.   Simple,  straight- 
forward statistical  measures  of  performance  are   exhibited  in  graphical 
and  tabular  form.   All  the  models  displayed  some  degree  of  difficulty  in 
tracking  particular  types  or  portions  of  anomalous  reliability  growth 
patterns.   The  cumulative  model  displayed  this  difficulty  the  least  and 
exhibited  good  variability  (precision)  performance  providing  confidence 
in  its  use.   The  instantaneous  model,  while  displaying  generally  good  ac- 
curacy, exhibited  poor  variability  performance.   Except  for  a  couple  of 
anomalous  situations,  the  discrete  model  showed  good  accuracy  and  vari- 
ability performance.   Forecasting  performance  of  all  the  models  proved 
to  be  worse  than  their  capability  to  determine  current  reliability 
status . 
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I  .   INTRODUCTION 

New  weapons  systems  components  characteristically  display  very  low 
initial  reliability  as  they  begin  progress  through  early  development  and 
testing.   This  low  infant  reliability  of  a  system  and  its  components  is 
common  to  many  commercial  systems  such  as  automobiles,  aircraft,  etc. 
as  well  as  complex  military  systems.   The  reticence  of  contractors  to 
divulge  early  test  data  that  reveals  this  low  infant  reliability  can 
handicap  the  project  officer  and  contractor  in  exercising  their  manage- 
rial responsibility  to  bring  the  system  in  on  time  at  the  specified  re- 
liability level.   Because  reliability  growth  is  not  tracked  in  early 
development  stages,  timely  opportunities  to  either  accelerate  or  inten- 
sify development  by  reallocating  funds  during  the  initial  design/devel- 
opment phases  of  a  system  acquisition  cycle  are  frequently  lost.   Conse- 
quently, managers  can  lose  an  opportunity  to  support  their  projects  at 
a  time  of  greatest  need. 

There  exi sts  an  abundance  of  reliability  growth  models  designed  to 
assist  managers  in  charting  the  reliability  status  of  a  system  and  its 
components  during  any  stage  of  development  and  testing.   Unfortunately, 
the  benefits,  accuracies,  and  comparisons  of  these  models  are   not  always 
understood  by  management  staff  who  could  profit  from  their  use. 

The  primary  purpose  of  this  thesis  was  to  evaluate  three  easily 
understood,  relatively  simple  reliability  growth  models  which  would 
hopefully  prove  to  be  reasonably  accurate  over  a  broad  spectrum  of 
reliability  growth  situations.   Conditioned  upon  this  demonstrated 
versatility  in  a  wide  variety  of  circumstances,  the  ultimate  goal  is 


for  employment  and  utilization  of  these  reliability  growth  models  by 
management  on  a  significantly  larger  scale  with  increased  willingness 
to  share  reliability  progress  information  between  contractor  and  custom- 
er.  The  evaluation  was  intentionally  not  structured  to  ascertain  the 
capabilities  of  the  reliability  growth  models  as  high  resolution  tools 
in  very  restricted  situations  but  rather  as  gross  models  for  flagging 
significant  deviations  of  a  system's  or  component's  reliability  progress 
from  a  desired  or  expected  reliability  growth  pattern.   Hence,  the 
models'  ability  to  alert  managers  that  consideration  of  corrective  action 
or  at  least  more  intense  scrutiny  of  reliability  status  is  required  was 
established  as  a  goal  of  the  evaluation.   Perhaps  the  greatest  potential 
windfall  benefit  of  utilizing  capable  reliability  growth  models  as  a 
project  management  tool  is  a  reduction  in  cost  overruns  that  have  plagued 
weapons  systems  development  since  the  1 950 '  s .   A  primary  cause  of  weapons 
systems  cost  overruns  as  noted  in  reference  3  is  trying  to  develop  and 
produce  the  systems  too  fast. 

Since  the  underlying  reliability  progress  pattern  of  a  new  system 
and  its  components  is  unknown  to  the  project  officer  and  contractor, 
these  managers  essentially  need  a  "model  for  all  seasons".   In  this  vein 
the  attributes  desired  in  a  reliability  growth  model  are   accuracy  and 
robustness.   Hence,  a  model (s)  that  can  discern  the  true  underlying  re- 
liability progress  pattern  with  some  degree  of  fidelity  (accuracy)  with- 
out sacrificing  the  capacity  to  cope  with  a  significantly  diverse  range 
of  reliability  progress  patterns  (robustness)  is  the  tool  managers  seek. 

With  these  attributes  in  mind  a  continuous  cumulative  failure  rate 
reliability  growth  model,  a  continuous  instantaneous  failure  rate  reli- 
ability growth  model,  and  a  discrete  reliability  growth  model  were  chosen 
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for  evaluation.  Since  each  model  is  structured  to  measure  a  different 
parametric  indicator  of  reliability  progress,  no  direct  comparisons 
between  the  models'  performance  were  made.   The  models  are   presumptuous 
as  the  generally  accepted  title  reliability  growth  models  implies.   In 
order  to  evaluate  the  models'  accuracy  and  robustness  not  only  were  they 
tested  against  "nice"  underlying  reliability  progress  paths,  but  also, 
they  were  confronted  with  underlying   patterns  which  exhibited  relia- 
bility stagnation  and  degradation  prior  to  continued  growth  as  well  as 
patterns  demonstrating  no  reliability  progress. 

The  three  reliability  growth  models  evaluated  in  this  thesis  are 
designed  to  be  very  general  in  the  nature  of  the  items  for  which  they 
propose  to  chart  reliability  progress.   The  models  can  be  employed  for 
reliability  tracking  at  the  system  level,  sub-system  level,  and/or  the 
individual  component  level  during  any  phase  of  an  acquisition  cycle  to 
include  even  retrofit  programs.   Hence,  for  the  balance  of  this  thesis 
"component"  and  "item"  are  used  to  reference  an  entire  system,  or  a 
sub-system,  or  an  individual  component  for  which  reliability  progress 
is  being  modelled  as  appropriate. 

The  evaluation  technique  utilized  computer  simulation  which  permits 
the  underlying  reliability  progress  patterns  to  be  specified  while 
observing  the  models'  performance  in  both  determining  and  forecasting 
the  underlying  progress  patterns.   The  simulation  technique  and  evalua- 
tion measures  are  described  in  the  next  chapter. 
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I  I .   EVALUATION  METHOD  AND  MEASURES  OF  EFFECTIVENESS 

A.   EVALUATION  METHOD 

Performance  evaluation  of  both  continuous  failure  rate  reliability 
growth  models  and  the  discrete  reliability  growth  model  was  accomplished 
using  computer  Monte  Carlo  simulation.   Computer  simulation  permits  the 
analyst  to  specify  and  control  the  underlying  reliability  path  of  growth. 
By  utilizing  simulation  data  that  is  equivalent  to  data  normally  avail- 
able in  a  testing  and  development  program  the  models'  performance  may 
be  compared  with  the  specified  underlying  reliability  path. 

The  continuous  cumulative  failure  rate  reliability  growth  model  and 
the  continuous  instantaneous  failure  rate  reliability  growth  model  both 
utilize  the  same  data  from  a  testing  and  development  program.   Therefore, 
these  two  models  were  evaluated  simultaneously  in  a  unified  simulation 
program.   The  test  program  assumed  for  the  continuous  models  is  essen- 
tially a  deterministic  program  in  which  testing  of  items  in  any  given 
test  phase  can  be  accomplished  simultaneously  since  the  number  of  items 
to  be  tested  within  each  phase  is  specified  prior  to  conduct  of  the 
test  phase.   On  the  other  hand  the  discrete  reliability  growth  model  is 
based  on  a  sequential  test  structure  in  which  the  number  of  units  tested 
cannot  be  specified  beforehand;  specifically,  testing  in  each  phase  is 
continued  until  a  specified  number  of  failures  are  observed  which  in 
turn  generate  a  significant  development  or  production  change  that  hope- 
fully improves  reliability.   Consequently,  the  number  of  tests  within  a 
test  phase  are  random.   Because  of  this  dissimilarity  in  test  program 
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structure  for  the  continuous  and  discrete  models,  a  separate  simulation 
program  was  run  for  evaluation  of  the  discrete  reliability  growth  model. 

B.   MEASURES  OF  EFFECTIVENESS 

As  indicated  in  the  introduction,  the  primary  goal  of  this  evaluation 
was  to  demonstrate  the  capability  of  selected  reliability  growth  models 
to  estimate  unknown,  underlying  reliability  progress  paths  with  suffi- 
cient accuracy  and  versatility.   Thus,  the  capability  of  the  models  to 
satisfy  these  requirements  is  a  question  of  their  accuracy  and  precision; 
i.e.,  (1)  how  close  do  the  models'  reliability  or  failure  rate  estimates 
approximate  the  true  underlying  reliability  or  failure  rate  paths  and 
(2)  are    the  models  consistent  in  their  approximations? 

To  answer  these  two  questions  regarding  the  models'  capabilities  the 
simulations  were  replicated  one-hundred  times  for  each  underlying  reli- 
ability progress  path  specified.   The  number  of  replications  was  actual- 
ly programmed  as  a  "user  specified"  input  variable  and  could  have  been 
chosen  to  be  ten  runs  for  one  progress  path  and  a  thousand  runs  for 
another  progress  path.   One-hundred  replications  for  all  progress  paths 
was  chosen  as  a  sufficiently  large  number  to  provide  confidence  in  the 
test  results  while  holding  computer  processing  time  requirements  within 
acceptable  limits.   Two  simple  statistics  were  then  computed  for  each 
model's  performance  on  each  specified  underlying  reliability  progress 
path. 

First,  the  average  values  (arithmetic  means)  of  the  models'  estimates 
of  the  parameter  utilized  in  the  characterization  of  the  underlying  re- 
liability progress  path  were  computed  to  provide  a  measure  of  the  models' 
accuracy.   Average  values  of  reliability  or  failure  rate  estimates  were 
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computed  for  both  the  current  reliability  status  at  the  end  of  a  test 
phase  and  the  forecast  reliability  status  at  the  end  of  the  following 
test  phase.   While  a  summary  of  these  parameter  mean  value  results  could 
be  presented  in  tabular  form,  such  mode  of  presentation  is  difficult  and 
time  consuming  to  comprehend.   Therefore,  a  graphical  presentation  was 
selected  over  the  tabular  form.   The  graphical  presentation  permits 
easy  qualitative  evaluation  of  a  model's  accuracy  and  recognition  of 
estimate  trends  such  as  consistent  optimistic/pessimistic  estimation  of 
the  reliability  progress  path. 

For  each  particular  reliability  growth  model,  test  structure,  and 
specified  underlying  reliability  progress  path,  a  graph  was  prepared 
depicting  the  true  underlying  path,  the  model's  mean  estimated  value  for 
the  reliability  or  failure  rate  at  the  end  of  each  test  phase,  and  the 
model's  mean  forecast  reliability  or  failure  rate  for  the  end  of  the 
following  test  phase.   Since  a  total  of  I98  different  combinations  were 
simulated,  graphs  for  all  the  simulation  results  are  not  presented.   In 
those  cases  where  no  significant  change  in  accuracy  or  trend  results 
occurs  graphs  are  omitted. 

Project  managers  are   concerned  with  only  one  "replication"  of  the 
reliability  progress  path  estimation  task  as  opposed  to  ten,  one-hundred, 
or  a  thousand  progress  path  estimates.   Given  that  the  mean  performance 
of  a  reliability  growth  model  is  satisfactory,  managers  need  to  know  if 
the  model  can  be  trusted  to  deliver  satisfactory  performance  on  that 
single  "replication".   This  concern  equates  to  the  question  of  varia- 
bility (precision)  in  the  model  estimates;  i.e.,  does  the  model  deliver 
"tight  groups"  around  the  mean  values  of  its  estimates?   To  measure 
variability  the  standard  deviation  of  each  model's  reliability  or  failure 
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rate  estimates  from  the  mean  value  performance  was  computed  for  all 
phases  of  all  simulations. 

Since  standard  deviation  is  a  relative  measure,  it  is  difficult  to 
recognize  trend  in  the  performance  of  the  reliability  growth  models  from 
a  table  of  standard  deviation  statistics.   So,  variability  performance 
of  the  reliability  growth  models  is  presented  in  tabular  form  containing 
entries  of  standard  deviation  of  the  model  estimates  expressed  as  per- 
centages of  the  magnitude  of  the  reliability  or  failure  rate  values  to 
which  they  correspond.   (This  statistic  is  the  coefficient  of  variation 
expressed  as  a  percentage;  i.e.,  the  percentage  standard  error.)   Because 
gross  performance  characteristics  of  the  reliability  growth  models  were 
being  evaluated,  all  percentages  were  truncated  to  the  whole  percentage 
point  to  facilitate  examination  of  the  variability  performance  tables 
for  trends. 

In  examining  the  variability  performance  of  the  reliability  growth 
models  some  guideline  is  desirable.   In  the  field  of  econometric  models 
a  percentage  standard  error  of  10-15%  is  the  goal  given  by  reference  6. 
In  cost  estimation  modelling  a  percentage  standard  error  of  15_20%  is 
considered  satisfactory  according  to  reference  8.   A  reliability  growth 
model  that  exhibits  good  mean  estimation  performance  but  displays  very 
poor  variability  performance  may  be  useless  for  practical  purposes.   On 
the  other  hand  a  reliability  growth  model  that  produces  excellent  vari- 
ability performance  coupled  with  poor  but  consistently  biased  mean  esti- 
mation performance  may  prove  to  be  very  useful.   Ultimately,  the  project 
manager  will  have  to  decide  if  the  performance  characteristics  displayed 
by  the  reliability  growth  models  examined  in  the  next  chapter  qualify 
these  models  for  application  to  his  project. 


III.   RELIABILITY  GROWTH  MODEL  DESCRIPTIONS 
AND  PERFORMANCE  RESULTS 

A.   CONTINUOUS  CUMULATIVE  FAILURE  RATE  RELIABILITY  GROWTH  MODEL 
1 .   Model  Description 

The  continuous  cumulative  failure  rate  model  evaluated  is  of 
the  form 

XTT  =  -S-  (3.1) 

TT   TTa 

where 

X_T  =  cumulative  failure  rate  of  the  item  thru  time  TT , 

TT  =  total  accumulated  test  time  for  all  testing,  and 

a,S  =  constants  that  determine  the  shape  of  the  model 
estimated  reliability  progress  path. 

Cumulative  failure  rate  may  also  be  thought  of  as  average  failure  rate 
or  characteristic  failure  rate.   Under  the  assumption  that  a   is  posi- 
tive, the  model  relationship  is  that  as  test  time  is  accumulated  on  an 
item  (and  by  implication  discrepancies  discovered  in  the  item  are   cor- 
rected) the  denominator  of  the  right-hand  side  of  equation  3-1  increases; 
and  hence,  the  cumulative  failure  rate  of  the  item  decreases.   So  reli- 
ability growth  is  modelled  as  decreasing  failure  rate  by  this  model. 
Total  accumulated  test  time  TT   is  measured  in  whatever  units  are   appro- 
priate to  the  item  being  tested;  i.e.,  minutes,  hours,  days,  years,  etc. 
or  mission  units.   The  model  is  considered  continuous  since  the  single 
independent  variable  TT  ,  total  accumulated  test  time,  is  continuous 
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as  opposed  to  a  discrete  variable  such  as  the  number  of  failures  experi- 
enced or  the  number  of  item  modifications  accomplished. 

If  the  failure  rate  being  experienced  during  any  particular 
phase  of  testing  is  thought  of  as  the  instantaneous  failure  rate,  then 
within  the  context  of  reliability  growth  instantaneous  failure  rate  is 
expected  to  decrease  as  testing  proceeds  from  phase  to  phase  just  as 
cumulative  failure  rate  is  expected  to  decrease.   Since  cumulative  fail- 
ure rate  is  a  characterization  of  the  failure  rate  of  the  item  based  on 
all  the  testing  that  has  been  accomplished;  i.e.,  an  "average"  measure 
of  failure  rate,  then  cumulative  failure  rate  is  expected  to  be  greater 
than  the  instantaneous  failure  rate  at  any  point  after  the  initial  test 
phase  when  the  two  measures  are  equal.   Conversely,  if  the  reliability 
of  the  item  decreases  as  the  acquisition  cycle  develops  (increasing  in- 
stantaneous and  cumulative  failure  rate),  then  cumulative  failure  rate 
is  expected  to  be  less  than  the  instantaneous  failure  rate  being  experi- 
enced at  any  phase  after  the  initial  phase  of  testing. 

A  mathematical  analysis  of  the  cumulative  failure  rate  model  is 
given  in  reference  2. 

2 .   Reliability  Testing  Procedure 

For  the  continuous  failure  rate  reliability  growth  models  the 
appropriate  reliability  testing  procedure  of  a  system  acquisition  model 
is  one  that  is  conducted  in  distinct  test  phases.   These  test  phases  are 
indexed  by  i  with  a  total  of  K  phases  comprising  the  duration  of  the  pro- 
cedure; i.e.,  i  =  1 ,  2 ,  3 ,  • • •  ,  K.  (Within  each  phase  of  a  testing  a 
specified  number  of  tests  are   conducted  where  the  number  of  tests  is 
denoted  by   NT.  .  \  The  number  of  tests  per  phase  NT.   can  vary  from 
phase  to  phase.   During  each  phase  of  testing  an  underlying,  inherent 
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failure  rate  that  is  unknown  to  the  project  managers/contractors  is  pres- 
ent in  the  item  under  test.   This  underlying  failure  rate  is  the  instan- 
taneous failure  rate  of  the  item  for  each  phase  and  is  denoted  as  X.    . 

i 

Also,  during  each  phase  of  testing  a  phase  total  test  time  is  accumulated 
and  labeled  as   T.  .   Hence,  at  the  completion  of  test  phase  i  the  total 

accumulated  test  time  TT.   on  the  item  is  given  by 

i 

i 
TT.  =  £  Tj  (3.2) 

J-1 

for  i  =  1,  2,  3,  ...,  K  and  TT  =  T   . 

For  each  phase  of  the  testing  program  a  planned  test  time  is  es- 
tablished beforehand  after  which  items  under  test  that  are   still  func- 
tioning are   judged  successful;  i.e.,  non-failures.   The  planned  test 
time  for  test  phase  i  is  denoted  as   PTT.  .   Therefore,  the  maximum  test 
time  that  can  be  accumulated  during  any  given  test  phase  i  is  given  by 

the  product  NT.  x  PTT.  and  occurs  only  when  none  of  the   NT.   components 
ii  i     r 

tested  fail.   During  each  phase  of  the  testing  the  number  of  failures 

that  does  occur  is  recorded  as  F-  .  [At  the  end  of  test  phase  i  the  total 

number  of  failures  experienced  for  the  entire  testing  procedure  thru 
phase  i  is  given  by 


(3-3) 


for  i  =  1 ,  2,  3,  • • • ,  K  and  TF  =  F   . 

Figure  3-1  is  a  schematic  diagram  of  the  continuous  models  reli- 
ability testing  procedure.   The  total  number  of  test  phases  K,  the  number 
of  tests  conducted  during  each  test  phase  NT.,  and  the  planned  test  time 
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PTT.   at  which  an  item  under  test  is  considered  successful  are  determined 

i 

prior  to  testing.   The  test  time  T.   accumulated  during  each  test  phase, 

\ 


the  total  accumulated  test  time 

number  of  failures   Fj   experienced  during  each  test  phase,  and  the  total 

number  of  failures   TF.   accumulate  for  the  entire  testing  procedure  thru 

the  end  of  each  test  phase  are   all  data  collected  from  the  reliability 

testing  procedure.   The  most  salient  feature  of  the  computer  simulation 

of  this  reliability  testing  procedure  is  that  the  analyst  also  specifies 

the  underlying  instantaneous  failure  rate   A.   that  is  effective  for  each 

phase.   Thus  an  accurate  assessment  of  the  accuracy  of  the  model  can  be 

made  because  the  A.   values  are  known. 

i 

3 •   Reliability  Testing  Procedure  Computer  Simulation 
a.   Summary 

The  reliability  test  procedure  appropriate  to  the  continuous 
failure  rate  reliability  growth  models  was  simulated  on  the  Naval  Post- 
graduate School  W.  R.  Church  Computer  Center's  IBM  360/67  System  utiliz- 
ing standard  computer  simulation  techniques.   Reliability  testing  proce- 
dure design  parameters  K,  NT.,  and  PTT.  were  specified  along  with 
various  underlying  instantaneous  failure  rate  sets.   Tests  of  components 
were  then  simulated  with  component  success  or  failure  being  determined 
for  each  test  based  on  the  appropriate  underlying  instantaneous  failure 

rate  A.   and  the  planned  test  time   PTT.  .   T. ,  TT. ,  F.,  and  TF.   test 
i  r  i      i     i    i         i 

data  were  collected  as  each  testing  procedure  simulation  was  accomplished 
Simulations  of  reliability  testing  procedures  for  each  specified  instan- 
taneous failure  rate  set  were  replicated  one-hundred  times  in  order  to 
assess  the  continuous  failure  rate  reliability  growth  models'  variability 
performance. 
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After  the  reliability  testing  procedure  simulations  were  com- 
pleted, continuous  failure  rate  reliability  growth  model  estimates  of  the 
underlying  cumulative  and  instantaneous  failure  rate  of  the  component 
being  tested  were  produced  based  on  the  simulation  test  data  by  computa- 
tions utilizing  ordinary  least  squares  regression  techniques.   Simple 
mean  and  variability  statistics  of  these  model  estimates  were  then  computed 
to  permit  performance  evaluation  of  the  continuous  failure  rate  models, 
b.   Detail  Description 

The  computer  simulation  of  the  continuous  cumulative  and  the 
continuous  instantaneous  failure  rate  reliability  growth  models  was  ini- 
tiated by  reading  in  the  following  reliability  testing  procedure  design 
specifications:   K,  the  total  number  of  test  phases;  NT.,  the  number  of 
tests  to  be  conducted  during  each  test  phase;  X.,    the  underlying  instan- 
taneous failure  rate  effective  during  each  test  phase;  and  NSIMS,  the 
number  of  times  the  specified  reliability  testing  procedure  was  to  be 
simulated.   For  this  thesis  NSIMS  was  always  specified  as  one-hundred 
s  imulat  ions . 

Rather  than  specify  the  test  phase  planned  test  times   PTT. 
at  which  components  still  operating  were  considered  not  to  have  failed 
for  each  phase,  planned  test  times  were  calculated  for  each  test  phase 
such  that 

exp  (-X.  x  PTT.)  =  0.7  or 

PTT.  =  -  j-   In  0.7  .  (3.*) 

i 

The  lifetimes  of  components  tested  during  each  test  phase  i  were  taken 

to  be  exponentially  distributed  with  mean  time  to  failure  (MTTF)  1/A.. 
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The  planned  test  times  given  by  equation  3-*+  result  in  each  component 
tested  having  a  ~JQ%   chance  of  surviving  the  test  phase.   This  figure  for 
survival  probability  of  0.7  was  chosen  arbitrarily  to  insure  that  some 
component  failures  were  generated. 

The  procedure  for  specifying  planned  test  times   PTT.   admit- 
tedly deviates  from  actual  test  procedure  design  since  the  calculation 
in  equation  3  •  *♦  is  dependent  upon  the  underlying  instantaneous  failure 
rate  which  is  unknown  in  real  life  testing.   However,  if  a  reliability 
testing  procedure  is  being  conducted  under  the  assumption  of  reliability 
growth,  planned  test  times   PTT.   would  naturally  be  specified  as  increas- 
ing from  test  phase  to  test  phase  as  the  expected  failure  rate  decreases. 
If  the  true  underlying  failure  rate  should  stagnate  or  worsen  from  one 
test  phase  to  the  next,  then  the  probability  that  components  fail  during 
the  next  test  phase  increases  since  planned  test  time  will  have  been 
lengthened.   Consequently,  more  failures  would  generally  result  than  if 
the  underlying  reliability  progress  path  had  followed  the  expected  "nice" 
growth  pattern;  i.e.,  decreasing  failure  rate. 

After  specification  data  were  read  into  the  computer,  uniform 

(0,1)  random  variates  were  drawn;  one  corresponding  to  each  component 

tested  in  each  of  the  K  total  test  phases;  i.e.,  uniform  random  variates 

were  drawn  and  indexed  as  U.  .  for  i  =  1 ,  2 ,  3 ,  • ■ •  ,  K  and  j  =  1,  2,  3, 

'  >  J 

...  ,  NT..   For  this  evaluation  the  life  length  of  components  was  assumed 
to  be  exponentially  distributed.   The  assumption  is  a  strong  one,  and  the 
performance  characteristics  of  the  models  under  other  than  the  exponen- 
tial lifetime  distribution  assumption  might  prove  to  be  a  fruitful  area 
for  evaluation.   Next,  the  uniform  random  variates  were  transformed  into 
exponential  lifetimes  for  the  components  under  test  via  the  probability 
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integral  transform  as  given  in  reference  h;    i.e.,  each  component  was  as- 
signed a  test  lifetime   E.  .   according  to 

i  ,J 


E..j--57,nU..j  (3-5) 


for  i  =  1,  2,  3,  ...  ,  K  and  j  =  1,  2,  3,  ...  ,  NT.. 

The  number  of  failures   F.   that  occurred  during  each  test 

phase  was  then  tallied  by  comparing  the  test  lifetime   E.  .   of  each  com- 

i  ,j 

ponent  with  the  appropriate  planned  test  time   PTT.   such  that 

NT.  (  1  if  E.  .  <  PTT.  (Fai lure) 

'  ,J      ' 


F. 

i 


-  /  j         5.  where  6 .  = <  (3-6) 


J-l 


0  if  E.  .  •  PTT.  (Success) 
i  ,J  =    ' 


for  i  =  1,  2,  3,  ...  ,  K.   Similarly,  the  test  time  T.   accumulated  for 
each  test  phase  was  recorded  as 

NT.  I  E.  .  if  E.  .  <  PTT. 

'  »J      '.J       i 


T. 
i 


-/]       e.  .  where  e.  .  =  j  (3-7) 

"    '.J        i, J   ] 

J-l 


PTT.  if  E.  .  >  PTT. 
i      i  ,j  =     i 

for  i  =  1,  2,  3,  •••  ,  K.   Finally,  the  corresponding  total  failures   TF 
and  total  test  times   TT.   accumulated  thru  the  end  of  each  test  phase 


were  tal 1 i  ed  as 


i 
TF.  =  ^    F.  ,  and  (3-8) 

J-l 

i 
TT.  =  Y        T.  (3-9) 

J-l 


for  i  =  1,  2,  3,  •••  ,  K  respectively. 
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In  keeping  with  accepted  reliability  growth  modelling  prac- 
tice for  reliability  testing  procedures  utilizing  planned  test  times, 
in  those  cases  where  no  failures  were  experienced  during  a  given  test 
phase  (F.  =  0)  the  number  of  failures  for  that  test  was  assigned  as  one- 
half  a  failure;  i.e.,  F.  =  —  .   Similarly,  whenever  the  total  failures 

.  i    2 

accumulated  thru  a  given  test  phase  were  zero  (TF.  =  0)  based  on  the 

i 
original  failures  per  test  phase  statistics  (TF.  =  £   F.  =  0),  then 

'    J-1    J 
total  failures  accumulated  thru  that  phase  were  assigned  as  one-half  a 

failure;  i.e.,  TF.  =  —  .   The  primary  thrust  of  these  modifications  is 
i    2 

that  zero  failure  statistics  result  in  failure  rate  estimates  of  0.0 
which  are  extremely  unrealistic  and  would  virtually  never  occur  under 
the  exponential  lifetime  distribution  assumption  if  infinite  planned  test 
times  were  possible.   Hence,  the  one-half  failure  assignment  procedure 
results  in  optimistic  but  positive  failure  rate  estimates  for  those  in- 
stances in  which  no  failures  are  experienced. 

These  data  constituted  the  total  data  gathered  from  a  single 
computer  simulation  of  the  continuous  model  reliability  testing  proce- 
dure for  a  single  specified  underlying  reliability  progress  path;  i.e., 
a  specified  set  of  failure  rates  which  shall  henceforth  be  referred  to 
as  a  lambda  set.   This  computer  simulation  was  repeated  one-hundred  times 
(NSIMS  =  100)  for  each  specified  lambda  set  where  the  simulation  repli- 
cation is  indexed  as  r  =  1,  2,  3,  •••  ,  NSIMS  =  100.   Therefore,  all  the 
simulation  data  are  indexed  by  the  simulation  number   r  which  has  been 
left  implicit  for  simplification;  i.e.,  the  simulation  results  are   F.   , 

TF.   ,  T.   ,  and  TT.    .   Since  twenty-eight  different  lambda  sets  were 
i  , r'   i  ,  r'  i , r  '        a 

utilized  for  evaluating  the  continuous  failure  rate  reliability  growth 
models  with  three  different  number  of  tests  per  test  phase   NT.   specified 
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for  each  lambda  set,  8400  reliability  testing  procedures  were  simulated 
for  the  continuous  reliability  growth  models  evaluation. 
k .      Computer  Simulation  Data  Manipulation 

The  continuous  cumulative  failure  rate  reliability  growth  model 
is  proposed  as  a  model  of  the  unknown,  underlying  cumulative  failure 
rate   AT_  of  a  component  as  it  proceeds  thru  a  system  acquisition  cycle. 
In  each  reliability  testing  procedure  computer  simulation  the  true  cumu- 
lative failure  rate  A_T   that  was  in  effect  at  the  end  of  a  test  phase 
was  determined  by  the  test  time  weighted  average  of  the  underlying  true 

instantaneous  failure  rates   X.   that  were  in  effect  thru  the  end  of  the 

i 

given  test  phase.   Thus,  the  true  cumulative  failure  rate  was  determined 
as 

1       T. 

'    J-1 

for  i  =  1,  2,  3,  •••  ,  K.   It  is  against  this  true  cumulative  failure 
rate  that  the  model's  performance  was  measured. 

At  the  end  of  a  phase  of  testing  an  estimator  of  the  instanta- 
neous failure  rate  for  that  phase  of  testing  is  given  by 

Number  of  Failures  Experienced  During  Phase  i    F. 

y^   a  =  _!_      (3.11) 

i    Test  Time  Accumulated  During  Phase  i  T. 

for  i  =  1,  2,  3,  •••  ,  K.   In  turn  an  estimator  of  the  cumulative  fail- 
ure rate  thru  the  end  of  a  phase  of  testing  is  given  by  the  test  time 
weighted  average  of  the  instantaneous  failure  rate  estimates;  i.e., 

^       !         T-         A 
TTi    ft    J   TTi 

for  i  =  1,  2,  3,  •••  ,  K.   Note  that  since  TT.  =  T  ,  then  AJT  =  A,  . 
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The  estimator  of  the  cumulative  failure  rate  thru  the  end  of  any  test 
phase  may  also  be  stated  as 

.       j-l       TF. 

^tt.  ■- *w:  (3.13) 

i  i 


E  Tj 


j  = 


for  1=1,2,  3,  •••  ,  K  by  the  following  line  of  reason:   F.  =  \.    T. 

J    J   J 


by  equation  3-11;  and  therefore, 


i  i 

TF.  =  J^        F-  =  J^  "^  T-   bV  equation  3-8. 
J-l        J-l 


So,  i 

*—         J  J    ; 

TF.    j  =  l  _    T. 

^tt.  -TT--TT: =E    Aj    xi:  (3-u) 

J-l 


for  i  =  1 ,  2 ,  3 ,  • • •  ,  K  wh  ich  is  equi val ent  to  equat  ion  3-12.   S  ince 
TF  =  F  and   TT  =  T   ,  then 

XTT1"TT""T~"  Xl   TT""  Xl   Tj""  Al  •  (3'15) 


The  estimates  of  the  instantaneous  failure  rate  \.      and  the 

i 

cumulative  failure  rate   att   are  based  on  data  from  a  reliability 

i 
testing  procedure  utilizing  planned  test  times   PTT.  .   Therefore,  the 

estimates  obtained  from  equations  3-11  and  3-13  sire   biased  estimates, 

the  bias  being  in  the  pessimistic  direction.   In  particular  all  of  the 
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instantaneous  failure  rate  estimates  and  the  cumulative  failure  rate 
estimates  for  the  early  phases  of  the  testing  procedure  are    biased  too 
large  in  magnitude.   To  compensate  for  this  estimator  bias  a  bias  correc- 
tion factor  derived  in  reference  7  was  applied  to  the  instantaneous  fail- 
ure rate  estimates   A.   and  the  cumulative  failure  rate  estimates  X. 

i 

such  that 


TT. 


F.  /   2NT.   \ 

Ai    -T:(t  +  2NT.  1    and  (3-l6) 


I 


^ 


(3.17) 


for  i  =  1,  2,  3,  ...  ,  K. 

Next,  estimates  of  ct  and   6   in  the  cumulative  failure  rate 
model,  equation  3-1,  were  obtained  using  ordinary  least  squares  regres- 
sion techniques.   Because  ordinary  least  squares  regression  requires  at 
least  two  observation  points,  estimates  for  a  and   3  could  only  be 
derived  from  the  model  starting  at  the  conclusion  of  phase  two;  i.e., 
estimates  were  derived  for  phases  1  =  2 ,  3 ,  ** ,  •••  ,  K. 

If  the  cumulative  failure  rate  model  in  equation  3-1  is  written 
as 

XTT  «  8.  TT.  ]  (3.18) 

TT.     i    i 

i 

and  the  logarithmic  transformation  is  applied  to  this  equation,  then 


In  X_  =  In  3.  -  a.  In  TT.  .  (3-19) 

TT .        ii       i 

i 
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Lett i  ng 


a.    =    In    3. 
i  i 


b.    =   -a. 


X      = 


■t  E   XJ 


,    and 


Y     = 


-t  £    Yj 


J" 


then    equation    3-19    is   of    the    form 


Y.    =   a.    +   b.    X 
i  i  i 


(3.20) 


Note  that  the  estimated  cumulative  failure  rates   \JT   from  equation 

i 
3-17  are   utilized  in  performing  the  regressions.   Applying  ordinary 

least  squares  regression  estimates  as  given  in  reference  6  the  'ih   and 

b.   estimates  are 

1 


Z(X.  -  X)Y. 
J      J 


i  =  l 


b .  =  — 


!       I 


and 


(3.21) 


2  (xj "  ~x) 


a^  =  Y  -  b^X 
1         1 


(3-22) 


for  i  =  2,  3,  ^ ,    •••  ,  K.   The  "0s.  and  "3^   estimates  are  then  obtained 

1  '1 

by  the  inverse  transformations  "a":  =  -b.   and  "8s!  =  exp  ( "a^  ).   Finally, 

t  1         1          1 
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these  a.   and   g.   estimates  are   utilized  in  the  cumulative  failure 
i        i 

rate  model,  equation  3-18,  to  produce  the  model  estimates  of  the  unknown, 
underlying  cumulative  failure  rate.   These  estimates  are   given  by 


TT. 


(3.23) 


at 


TT. 


for  i  =  2,  3,  **,  •••  ,  K.   For  the  first  test  phase  the  cumulative  fail- 
ure rate  estimate  is  simply  taken  as  the  estimate  given  by  equation  3-17; 
i  .e.  , 

-"*      -       -     F,   /   2  NT,   \ 

(3.24) 


Again,  since  the  reliability  testing  procedure  was  simulated  one- 
hundred  times  (NSIMS  =  100)  for  each  lambda  set,  one-hundred  estimates 
of  the  cumulative  failure  rate  were  obtained  for  each  test  phase.   The 
mean  value  and  standard  deviation  of  these  estimates  were  computed  as 


1 


TT.    NSIMS 


NSIMS 

X 

r=1 
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for  i  =  1,  2,  3,  ••.  ,  K.   Finally,  the  measure  of  variability  for  the 
cumulative  failure  rate  model  was  computed  as 

S.D  " 


X 


P.S.E 


X 


TT. 


TT 


TT. 


x  100 


(3-27) 
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for  i  =  1,  2,  3,  ...  ,  K. 

Because  the  phase  total  test  time  T.   and  the  total  accumulated 

i 

test  time  TT.  are    random  variables  which  chanqed  from  simulation  to 
simulation,  their  mean  values  were  computed  as 

NS  IMS 

~  =  ucTmc  Y*        T-     and  (3-28) 

i     NS  I  MS   Z_/    i  ,  r 

r=l 

NSIMS 

Wl  =NSTSS   £   TTi,r  (3-29) 

r=1 

respectively  for  i  =  1,  2,  3,  •••,  K.   Similarly,  the  mean  value  of  the 

true  underlying  test  time  weighted  average  cumulative  failure  rate  was 

computed  as 

NSIMS 


TT.    NSI 


MS   E    XTT.  (3-30) 


i  :      i  ,  r 

r=l       ' 

for  i  =  1,  2,  3,  .••  ,  K.   Figure  3-2  is  a  graphical  representation  of 
the  results  obtained  from  the  reliability  testing  procedure  for  the  con- 
tinuous cumulative  failure  rate  reliability  growth  model. 

Equations  3-25  (  XJT    )    and  3-7  (P.S.E   *   )  are  measures  of  the 

i  TT . 

i 

cumulative  failure  rate  reliability  growth  model's  performance  in  deter- 
mining what  a  component's  reliability  status  is  at  the  end  of  a  test 
phase.   To  examine  the  cumulative  model's  forecasting  capability  the 

following  procedure  was  followed.   The  total  accumulated  test  time  TT. 

i 

of  the  cumulative  model,  equation  3-18  ,  is  a  stochastic  (random)  vari- 
able. Hence,  at  the  end  of  any  test  phase  i  the  total  accumulated  test 
time  for  subsequent  phases  of  testing  (TT.  .,  TT.   ,  ,..)  is  unknown  but 
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required  if  the  model  is  to  be  used  for  forecasting.   For  this  evalua- 
tion a  one  test  phase  in  the  future  forecast  of  cumulative  failure  rate 
was  made  by  approximating  the  next  test  phase  (i+l)  total  accumulated 
test  time  from  the  current  test  phase  as 

TTi+1  -  TT,  ♦  (NT.+]  x  PTT.+1)  (3.31) 

for  i  =  2,  3,  ^,  . . .  ,  K.   Thus  at  the  end  of  a  given  test  phase  i  the 

recorded  total  accumulated  test  time  TT.   for  that  phase  was  added  to 

the  maximum  phase  total  test  time  that  could  be  accumulated  during  the 

following  test  phase  i+l;  i.e.,  the  number  of  items  to  be  tested   NT. , . 

i  +  l 

times  the  planned  test  time   PTT. , .   for  the  next  test  phase.   This  pro- 

t  +  1 

cedure  thus  approximated  the  most  optimistic  outcome;  i.e.,  no  failures 

for  the  following  test  phase  i+l.   The  approximated  total  accumulated 

test  time  TT.  ,   for  the  next  phase  was  then  utilized  in  the  current 
i  +  l 

phase  i  cumulative  model  to  obtain  a  next  phase  forecast  cumulative 
fai 1 ure  rate  as 

I  +  l 

for  i  =  2,  3,  4,  ...  ,  K.   Therefore,  forecast  cumulative  failure  rates 
were  made  for  test  phases  i  =  3 ,    ** ,  5 ,  •  •  •  ,  K.   No  forecast  could  be 
made  for  test  phase  2  because  the  model  parameters   a.   and   ft.   were  not 
estimated  until  test  phase  2  was  completed.   No  forecast  was  made  past 
test  phase  K  since  there  would  be  no  underlying  cumulative  failure  rate 
with  which  to  compare.   Finally,  forecast  cumulative  failure  rate  sta- 
tistics similar  to  the  determined  failure  rate  statistics  in  equations 
3-25,  3-26,  and  3-27  were  computed  for  the  simulation  replications 
(NSIMS  =  IOC)  as 
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NSIMS 

2 


FXTT    \  NSIMS  -  1 

TT .   T  *—?        i  ,  r 

i  r=l 


FATT. 


£    (FXTT<    -  FXJT  )Z  ,  and       (3.3*0 


P.S.E.^    =   — - -  x  100  (3-35) 

TT.      <s 

FXTT. 


for  f  -  3,  *»,  5 K. 

As  a  starting  point  for  the  evaluation  of  the  continuous  failure 
rate  models  the  lambda  sets  utilized  by  Jayachandran  and  Moore  in  refer- 
ence 5  were  adopted  as  specified  underlying  instantaneous  failure  rate 
X.   sets  for  five  reliability  progress  paths.   These  lambda  sets  are 
listed  in  table  3-1  where  they  are   numbered  as  lambda  sets  1  thru  5- 
Lambda  sets  1  thru  5  represent  "nice"  reliability  progress  paths  since, 
although  they  vary  in  rate  and  span  of  failure  rate  decay,  they  have 
smooth  exponential  like  decay  patterns  (curves).   A  decaying  failure  rate 
path  is  equivalent  to  a  reliability  progress  path  that  displays  growth. 

Note  in  table  3-1  that  for  lambda  set  3  for  example  the  specified 
underlying  instantaneous  failure  rate  starts  at  0.700  for  test  phase  1, 
decreases  to  0.3530  during  test  phase  2,  and  finally  reaches  a  value  of 
0.0500  for  the  last  test  phase,  test  phase  16. 

Because  a  primary  goal  of  this  thesis  was  to  evaluate  the  robust- 
ness of  the  selected  reliability  growth  models,  the  "nice"  lambda  sets  1 
thru  5  were  augmented  with  nine  anomalous  lambda  sets  numbered  6  thru  ]h 
which  are  also  listed  in  table  3.1.   Lambda  sets  6  and  7  represent 
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approximately  linear  underlying  reliability  progress  paths.   Lambda  sets 
8,  9,  and  10  portray  underlying  reliability  progress  paths  in  which 
reliability  rapidly  attains  a  certain  level  and  stagnates  at  that  level 
for  the  duration  of  the  reliability  testing  procedure  (acquisition  cycle). 
Lambda  sets  11  and  12  display  initial  "nice"  reliability  growth  followed 
by  a  period  of  reliability  stagnation  and  a  subsequent  continued  growth 
pattern.   Finally,  lambda  sets  13  and  ]k    represent  underlying  reliability 
progress  paths  in  which  reliability  growth  is  interrupted  by  a  period  of 
reliability  degradation. 

As  indicated  in  table  3-1  lambda  sets  1  thru  1 k   each  represents 
reliability  testing  procedures  of  systems  acquisition  cycles  consisting 
of  sixteen  test  phases  (K  =  16).   For  very  expensive  and  complex  weapons 
systems  a  sixteen  test  phase  procedure  may  not  be  feasible.   Therefore, 
lambda  sets  for  a  more  realistic  six  test  phase  (K  =  6)  cycle  were  also 
utilized  in  the  computer  simulation  evaluation.   These  lambda  sets  for  a 
contracted  reliability  testing  procedure  are  listed  in  table  3-2  and 
numbered  MODI  thru  MOD  14.   Lambda  sets  MODI  thru  M0D14  represent  underly- 
ing reliability  progress  patterns  that  are  equivalent  to  the  progress 
patterns  characterized  by  lambda  sets  1  thru  ]k   but  which  develop  during 
a  six  test  phase  cycle  rather  than  over  a  sixteen  test  phase  cycle. 

In  order  to  examine  the  continuous  failure  rate  reliability 
growth  models  for  a  representative  set  of  acquisition  programs,  procedures 
with  items  tested  per  test  phase  NT.   of  five,  ten,  and  twenty  items  were 
simulated  utilizing  all  twenty-eight  lambda  sets;  i.e.,  NT.  =  5,  10,  and 
20  for  i  =  1,  2,  3,  •••  ,  16  or  i  =  1,  2,  3,  4,  5,  6  as  appropriate. 
Hence,  28  x  3  x  100  =  8400  reliability  testing  procedures  were  simulated 
for  the  continuous  failure  rate  reliability  growth  models  evaluation. 
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5.   Model  Performance 

a.   Accuracy  Performance 

Figures  3-3  thru  3-^1  present  selected  cases  of  the  continu- 
ous failure  rate  reliability  growth  model's  capability  to  determine  and 
forecast  the  unknown,  underlying  cumulative  failure  rate  progress  path 
during  an  acquisition  cycle.   These  graphs  depict  the  mean  test  time 

weighted  average  true  underlying  cumulative  failure  rate  \JT        progress 

i 
path  from  equation  3-30  ( ,  solid  line),  the  mean  model  determined 

cumulative  failure  rate   A_T   for  each  test  phase  from  equation  3-25  (0, 

circles),  and  the  mean  model  forecast  cumulative  failure  rate  FA--   f°r 

i 
the  third  thru  the  final  test  phase  from  equation  3-33  (X  ,  crosses) 

plotted  versus  the  mean  total  accumulated  test  time  TT.   from  equation 
3.29.   Note  that  the  point  plotted  for  the  test  phase  1  mean  model  deter- 
mined cumulative  failure  rate  is  not  a  mean  model  determined  estimate; 
but  rather,  it  is  the  mean  estimate  of  the  cumulative  failure  rate  (and 
instantaneous  failure  rate  since  the  estimate  is  for  test  phase  1)  given 

by  equation  3-24  (  X__  =  A.).   So  the  accuracy  of  the  reliability  esti- 

1 
mator  of  equation  3-2^  may  also  be  observed  to  a  degree  by  examining  the 

results  for  test  phase  1. 

To  illustrate  these  quantities  graphed  in  the  accuracy  perfor- 
mance figures  note  on  figure  3-11  that  at  approximately  82.00  total  accu- 
mulated test  time  units  the  mean  weighted  cumulative  failure  rate   ATT 

X"l6 
was  0.30  while  the  mean  model  determined  cumulative  failure  rate  XtT 

fTl6 
was  approximately  0.^5  and  the  mean  model  forecast  cumulative  failure 

rate   FATT    was  approximately  0.50.   Since  these  values  were  the  last 

1  'l6 
points  on  the  plot,  they  represent  the  simulation  results  for  the  six- 
teenth phase  of  a  reliability  testing  procedure  wherein  five  tests  per 
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phase  were  conducted  and  for  which  lambda  set  7  from  table  3-1  specified 
the  unknown,  underlying  reliability  progress  path.   Also,  note  that  the 
first  mean  model  forecast  cumulative  failure  rate  point  corresponds  to 
the  third  mean  model  determined  cumulative  failure  rate  plotted;  i.e., 
the  first  model  forecast  cumulative  failure  rate  was  for  test  phase  3 
(i  =3)-   Finally,  for  test  phase  1  the  point  plotted  for  the  mean  model 
determined  cumulative  failure  rate  is  approximately  0.73  which  is  actual- 
ly the  mean  value  of  the  estimate  produced  by  the  instantaneous  failure 
rate  estimator  of  equation  3  -  2^+ .   Notice  that  the  mean  test  time  weighted 
average  cumulative  failure  rate  at  test  phase  1  is  0.70  which  corresponds 
to  the  specified  underlying  instantaneous  failure  rate  for  this  phase 
from  table  3-1-   This  equivalence  is  explained  by  equation  3-15. 

Figures  3-3  thru  3.9  and  3.25  thru  3-29  indicate  that  for 
"nice"  underlying  reliability  progress  paths  the  cumulative  failure  rate 
model  displays  very  accurate  performance  with  slight  pessimistic  bias 
(higher  failure  rate  estimates).   "Least  data"  graphs;  i.e.,  five  tests 
per  phase  cases  (NT.  =  5)  are   generally  displayed.   Ten  and  twenty  tests 
per  phase  (NT.  =  10/20)  performance  was  as  good  as  or  better  than  five 
tests  per  phase  for  all  cases  in  which  only  the  NT.  =  5  graphs  are 
shown.   This  consistency  of  performance  is  displayed  for  lambda  set  3, 
NT.  =  5/10/20  sequence  in  figures  3-5,  3-6,  and  3-7-   Forecasting  perfor- 
mance displays  a  general  trend  of  overly  pessimistic  forecasts  for  the 
first  two  or  three  phases  improving  rapidly  after  four  or  five  forecasts 
have  been  made. 

Figures  3-10  thru  3-13,  3-30,  and  3-31  reveal  that  the  cumu- 
lative failure  rate  model  has  some  difficulty  in  tracking  linear  under- 
lying reliability  growth  patterns  even  when  a  relatively  large  amount 
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of  data  are  available  (figure  3-13,  NT.  =  20).   In  all  the  linear  cases 
both  the  model  determined  and  model  forecast  mean  failure  rates  diverge 
from  the  actual  cumulative  failure  rate.   Performance  does  not  improve 
as  the  number  of  components  tested  per  phase  is  increased  as  evidenced 
by  the  lambda  set  7,  NT.  =  5/10/20  sequence  performance  in  figures  3-11, 
3.12,  and  3-13. 

Figures  3-l/+  thru  3.20  and  3.32  thru  3-36  indicate  that  the 
cumulative  failure  rate  model  copes  with  permanently  stagnated  underly- 
ing reliability  progress  paths  easily.   The  figure  series  3-1^  thru  3-16, 
3.18  thru  3-20,  and  3-32  thru  3-3^  contrast  the  accuracy  performance  for 
increasing  number  of  test  per  phase  (NT.  =  5/10/20).   Again,  forecasting 
performance  is  initially  very  poor  but  quickly  improves  after  three  of 
four  forecasts  have  been  made. 

In  situations  of  the  underlying  reliability  progress  path 
experiencing  temporary  stagnation  figures  3-21,  3-22,  and  3-37  thru  3-39 
reveal  that  the  cumulative  failure  rate  model,  while  picking  up  the 
stagnation  credibly,  has  difficulty  in  accurately  modelling  the  cumula- 
tive reliability  improvement  following  the  period  of  temporary  stagna- 
tion.  During  this  post  stagnation  improvement  phase,  the  cumulative 
model's  determined  and  forecast  failure  rate  estimates  diverge  on  the 
pessimistic  side  from  the  true  underlying  cumulative  failure  rate  path 
much  as  they  did  in  the  linear  underlying  progress  path  cases.   Figures 
3.38  and  3-39  show  that  this  trait  is  not  relieved  by  gathering  more 
test  data  per  test  phase  of  the  reliability  testing  procedure  (NT.  =  5 
vs.  NT.  =  20)  . 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  1:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  2:  16  PHASES,  5  TESTS/PHASE 
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FIGURE    3.5 


CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  3:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  3:  16  PHASES,  10  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  5:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  7:  16  PHRSES.  20  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  8:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  8:  16  PHASES,  20  TESTS/PHASE 
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LAMBDA  SET  9:  16  PHASES.  5  TESTS/PHASE 


5k 


i  MEAN  HEIGHTED  CUMULATIVE  FAILURE  ROTE 
,  HERN  MODEL  DETERMINED  CUMULATIVE  FAILURE  RATE 
»  MEAN  MODEL  FORECAST  CUMULATIVE  FAILURE  RATE 


130.00    195.00    260.00    325.00    3AO.0O    455.00 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE    3.  18 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  10:  16  PHASES,  10  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  11:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  12:  16  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  MODI:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAMBDA  SET  M0D2:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  MODS:  6  PHASES,  5  TESTS/PHASE 


66 


o 
o 


a 
en 


.  i  MEAN  WEIGHTED  CUMULATIVE  FAILURE  RATE 
O  «  MEAN  MOOEL  DETERMINED  CUMULATIVE  FAILURE  RATE 
X  t    MEAN  MODEL  FORECAST  CUMULATIVE  FAILURE  RATE 


a 

GO 


o 


cc 
o 

UJ<D 

a:-- 

3° 


Q 


a: 


o 
in 

LU   -. 

>° 

•— * 
\— 
CC 

_] 

O 

CE 
li  lO 

21^ 


o 


o 
o" 


0 


o 
o 


^)',oo         3.00         5.00         9'.  00         12.00       7¥7oo~ 

TOTAL  ACCUMULATED  TEST  TIME 


18.00 


21.00 


FIGURE    3.30 


CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D6:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  M0D7;  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D8:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAM80A  SET  M0D8:  S  PHASES,  10  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D8:  6  PHASES,  20  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M009:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.36 

CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D10:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  MOO  1  1  :  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D12:  6  PHASES,  5  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAMB0A  SET  M0012:  6  PHASES,  20  TESTS/PHASE 
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CUMULATIVE  RELIABILITY  GF0WTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D13:  6  PHASES,  5  TESTS/PHASE 


77 


o 
o 


o 


_  t  MEAN  WEIGHTED  CUMULATIVE  FAILURE  ROTE 
O  ,  MEAN  MODEL  DETERMINED  CUMULATIVE  FAILURE  RATE 
X 


:  MEAN  MODEL  FORECAST  CUMULATIVE  FAILURE  RATE 


O 

CO 


O 

UJ°' 

I— 

cc 
<r 


o 

ID 


CE 
ii  lO 

zzco 


a 

CVJ 


o 


o 

o 


^D.OO  7.00  14.00  21.00  28.00  3^.00 

TOTAL  ACCUMULATED  TEST  TIME 


>42.00 


U9.00 


FIGURE    3.41 

CUMULATIVE  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAMB0A  SET  MOOIM:  6  PHASES,  5  TESTS/PHASE 
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Figures  3-23,  3.24,  3-^0,  and  3-^1  indicate  that  in  instances 
where  reliability  growth  progress  patterns  are    interrupted  by  a  period  of 
reliability  degradation  the  cumulative  failure  rate  model  lags  behind  the 
true  underlying  cumulative  failure  rate  path  with  both  its  determined  and 
forecast  failure  rate  estimates.   While  this  characteristic  might  be  dis- 
missed as  the  result  of  the  "smoothing"  nature  of  the  model,  the  model's 
failure  rate  estimates  are  plotted  with  the  mean  test  time  weighted  aver- 
age cumulative  failure  rate  which  is  itself  a  "smoothed"  quantity.   Hence, 
the  model  cannot  be  excused  from  failing  to  accurately  portray  the  tem- 
porary reliability  progress  decay  case.   "Least  data"  cases  (NT.  =  5)  are 
presented  and  the  performance  was  no  worse  for  the  ten  and  twenty  test 
per  phase  simulations.   The  cumulative  model  failure  rate  estimates  did 
display  reliability  progress  patterns  that  resembled  dampened  versions  of 
the  true  underlying  cumulative  failure  rate  paths,  and  the  model's  esti- 
mates were  reasonably  accurate  for  the  final  cumulative  failure  rate 
status. 

b.   Variability  (Precision)  Performance 

Tables  3-3  thru  3-8  present  the  continuous  cumulative  failure 
rate  reliability  growth  model's  variability  performance  for  determining 
the  cumulative  failure  rate  status  of  a  component  in  a  systems  acquisi- 
tion cycle.   Entries  in  these  tables  (P.S.E.<">  )  were  determined  by 

TTi 
equation  3-27.   Tables  3-9  thru  3-14  present  the  cumulative  failure  rate 

model's  variability  performance  for  forecasting  the  cumulative  failure 

rate  status  of  a  component  one  test  phase  ahead  (i+1)  of  the  current  test 

phase  (i).   Entries  in  these  tables  (P.S.E/£>   )  were  determined  by 

FATT. 
equation  3-35. 
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From  figure  3-11  the  mean  model  determined  cumulative  failure 

rate  for  the  sixteenth  phase  of  testing  for  lambda  set  7  using  five  tests 

per  phase  was  read  as  Ail-   =  0.45.   In  table  3-3  under  lambda  set  7, 

M16 

phase  16  a  value  of  25%  is  entered  for  P.S.E/T3?   •   Therefore,  the 

TT 
16 
standard  deviation  of  the  mean  model  determined    cumulative  failure 

rate  XI-        =   0.^5  over  one-hundred  simulations  (NSIMS  =  100)  was 
TT16 

P .  S  .  E  .*T> 

s'D-1cT  -  ?t„  *  ioo  TT'&  '°^5xT5o'  -1125    (3'36) 

TT,6      16 

from  equation  3-27. 

As  noted  in  Chapter  II  econometr i ci ans  and  cost  estimators 
set  goals  of  10%  to  20%  for  the  percentage  standard  error  measure  of  vari- 
ability performance  of  their  models.   Since  gross  performance  is  being 
examined  in  this  thesis,  less  stringent  bench  marks  may  be  appropriate 
for  evaluating  the  variability  performance  of  the  reliability  growth 
model s. 

If  a  goal  of  30%  is  set  for  the  percentage  standard  error 
measure  of  variability  performance,  then  a  quick  method  of  examining  a 
model's  variability  performance  is  to  determine  in  each  variability  per- 
formance table  the  first  test  phase  at  which  the  percentage  standard 
error  for  all  lambda  sets  (failure  rate  sets)  meets  the  30%  goal.   The 
earlier  the  test  phase  number  at  which  this  criterion  is  achieved  the 
more  confidence  can  be  placed  in  the  precision  of  the  model.   As  with 
any  sampling  statistic,  increasing  sample  size  is  expected  to  reduce 
variability.   In  the  reliability  testing  procedure  as  the  number  of  tests 
per  phase  NT.  is  increased  from  five  to  ten  to  twenty  the  test  phase 
number  at  which  the  30%  percentage  standard  error  goal  is  achieved  is 


80 


therefore  expected  to  occur  earlier  in  testing. 

Tables  3-3  thru  3-1^  list  the  variability  performance  of  the 
continuous  cumulative  failure  rate  reliability  growth  model.   In  tables 
3-3>  3-^>  and  3-5  the  cumulative  failure  rate  model's  variability  perfor- 
mance for  determined  cumulative  failure  rate  and  s ixteen  phase  lambda  sets 

achieves  the  30%  goal  at  phase  10  for  NT.  =  5,  at  phase  6  for  NT.  =  10, 

i  i 

and  at  phase  2  for  NT.  =  20.   The  percentage  standard  error  listed  for 
phase  1  in  any  of  the  tables  is  for  the  failure  rate  estimates  given  by 
equation  3-2^  vice  for  the  cumulative  model  estimates  which  start  with 
test  phase  2.   As  can  be  seen  the  equation  3-24  estimates  are   subject  to 
a  great  deal  of  variation.   Tables  3-6,  3-7,  and  3-8  show  that  in  the 
case  of  the  contracted  six  test  phase  cycles  the  cumulative  model's  vari- 
ability performance  for  determined  cumulative  failure  rate  fails  to 
achieve  the  30%  goal  for  NT.  =  5  and  achieves  the  30%  goal  at  phase  5 
for  NT.  =  10  and  phase  3  for  NT.  =  20. 

In  tables  3-9,  3-10,  and  3-11  the  cumulative  failure  rate 
model's  variability  performance  for  forecast  cumulative  failure  rate  in 
sixteen  test  phase  cycles  achieves  the  30%  percentage  standard  error 
goal  at  phase  12  for  NT.  =  5,  at  phase  7  for  NT.  =  10,  and  at  phase  5  for 
NT.  =  20.   Of  course  forecasts  are  made  for  phases  3  and  on  only.   Tables 
3-12,  3-13,  and  3-1^  show  that  for  the  six  phase  lambda  sets  the  cumula- 
tive model  fails  to  achieve  the  30%  percentage  standard  error  goal  for 
variability  performance  in  forecasting  cumulative  failure  rates.   The 
difficulties  arise  for  lambda  sets  M0D7,  M0D12,  and  M0D14.   These  are    the 
lambda  sets  for  which  the  mean  test  time  weighted  average  underlying 
cumulative  failure  rate  decreased  most  rapidly  to  approximately  0.20  for 
phase  6  in  all  cases,  and  these  lambda  sets  are  the  ones  for  which  the 


81 


cumulative  model  displayed  the  most  difficulty  in  tracking  the  underly- 
ing cumulative  failure  rate  progress  path.   Refer  to  figures  3-31,  3-39, 
and  3.41.   Note  in  table  3.\k    (NT.  =  20)  that  the  30%   goal  was  achieved 
in  phase  5  but  lost  in  phase  6  for  the  three  lambda  sets  enumerated 
above.   If  lambda  sets  M0D7,  MOD  1 2 ,  and  M0D1*4  were  not  considered,  then 
the  30%  percentage  standard  error  goal  was  achieved  in  phase  6  for  NT.  = 
10. 

B.   CONTINUOUS  INSTANTANEOUS  FAILURE  RATE  RELIABILITY  GROWTH  MODEL 
1 .   Model  Description 

The  continuous  instantaneous  failure  rate  reliability  growth 
model  evaluated  is  of  the  form 

X  =  ^f^  (3.37) 

TTa 

where 

X  =  instantaneous  failure  rate, 

TT  =  total  accumulated  test  time  for  all  testing,  and 

a,b  =  model  parameters  that  determine  the  shape  of  the  model 
estimated  instantaneous  failure  rate  reliability 
progress  path. 

Total  accumulated  test  time  TT,  the  independent  variable,  is  the  same 
quantity  utilized  in  the  continuous  cumulative  failure  rate  reliability 
growth  model,  equation  3-1 •   Instead  of  the  cumulative  failure  rate 
X__  modelled  by  equation  3.1,  the  instantaneous  model  of  equation  3-37 
models  the  unknown,  underlying  instantaneous  failure  rate  X.  .   There- 
fore, rather  than  producing  estimates  of  the  underlying  test  time 

weighted  average  cumulative  failure  rate  X__   ,  the  instantaneous  model 

i 
was  used  to  estimate  the  actual  test  phase  instantaneous  failure  rates 
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X.      specified  as  inputs  to  the  reliability  testing  procedure  computer 
simulation.   These  are  the  failure  rates  of  lambda  sets  contained  in 
tables  3- 1  and  3-2. 

As  can  be  seen  in  equation  3-37  and  similar  to  the  cumulative 
failure  rate  model,  as  total  accumulated  test  time  increases  instanta- 
neous failure  rate  is  modelled  as  decreasing.   Thus,  reliability  of  com- 
ponents is  portrayed  as  increasing  in  general.   Assumptions  regarding 
the  signs  of  the  values  for  the  model  parameters  a  and  b  in  the  instan- 
taneous model  are  not  as  obvious  as  the  assumptions  might  be  for  the 
signs  of  the  model  parameters  a  and  6  in  the  cumulative  failure  rate 
model . 

A  mathematical  analysis  of  the  instantaneous  failure  rate  model 
is  given  in  reference  2.   A  Monte  Carlo  simulation  evaluation  and  com- 
parison of  the  instantaneous  failure  rate  model  is  contained  in  refer- 
ence 5- 

2 .  Reliability  Testing  Procedure 

The  reliability  testing  procedure  of  a  system  acquisition  cycle 
appropriate  to  the  continuous  instantaneous  failure  rate  reliability 
growth  model  is  identical  to  the  procedure  described  for  the  cumulative 
failure  rate  model  in  chapter  II  I. A. 2.   Therefore,  the  test  data  neces- 
sary for  evaluation  of  the  instantaneous  model  were  collected  simul- 
taneously with  the  collection  of  data  for  the  evaluation  of  the  cumula- 
tive model  in  a  reliability  testing  procedure  as  graphically  depicted  in 
figure  3-1.   No  additional  test  results  are  required  for  evaluation  of 
the  instantaneous  failure  rate  model. 

3 .  Reliability  Testing  Procedure  Computer  Simulation 

Since  the  reliability  testing  procedure  applicable  to  the  con- 
tinuous and  instantaneous  failure  rate  models  are    identical,  data  for 
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evaluating  both  models  were  collected  from  a  single  computer  simulation 

program  as  described  in  chapter  I  I  I. A. 3-   Features  of  the  reliability 

testing  procedure  computer  simulation  such  as  phase  planned  test  time 

PTT.   determination  and  assignment  of  one-half  a  failure  in  those  i n- 
i 

stances  when  no  failures  occurred  during  a  test  phase  were  in  affect 
for  the  evaluation  of  botti  the  failure  rate  models.   Also,  the  same 
twenty-eight  lambda  sets  (failure  rate  sets)  given  in  tables  3-1  and 
3.2  and  the  same  number  of  simulation  replications  ( NS I  MS  =  100)  were 
utilized  for  both  model  evaluations. 

h .      Computer  Simulation  Data  Manipulation 

The  continuous  instantaneous  failure  rate  reliability  growth 
model  of  equation  3-37  may  be  written  as 

X.   =  (l-a.)b.(TT.)   '  .  (3.38) 

As  noted  in  equation  3-16  an  estimate  of  the  unknown,  underlying  instan- 
taneous failure  rate  of  a  component  for  any  given  test  phase  of  a  reli- 
ability testing  procedure  corrected  for  planned  test  time  bias  is  given 
by 

/\     2NT.     F. 

Ai  =  1  +  2NT.  '  IT  (3'39) 

1     1 

for  i  =  1,  2,  3,  •••  ,  K  where  T.   is  the  total  test  time  accumulated 

for  the  number  of  components   NT.   tested  during  the  phase  and   F.   is 

the  number  of  component  failures  occurring  during  the  phase. 

If  the  logarithmic  transformation  is  applied  to  equation  3-38, 

then 

In  X.    =  In  (l-a.)b.  -  a. In  TT.  .  (3-^0) 

1  1   1     1      1 
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To  obtain  the  ordinary  least  squares  regression  estimates  for  a.   and 
b.   the  data  pairs  (In  X.  ,    In  TT.)  are  employed  from  the  results  of  the 
reliability  testing  procedure  computer  simulation.   Let 


Y.  =  In  X.     , 
i         i 


X.  =  In  TT.  , 
i         i 


Y.  = 

i 


i 
1 


j-i 


i    ■ 

x.  =  -  V   x. 
J-i 


for    i    =    1 ,    2 ,    3 ,    • • •     ,    K.      Then   equation   3-^0   becomes 

Y.    -    In   b.(l-a.)    -    a.X.  (3-M) 

i  ii  ii 

for  i  =  1,  2,  3,  ...  ,  K.   By  the  derivation  presented  in  Appendix  A  the 

ordinary  least  squares  regression  estimates  for  the  instantaneous  failure 

rate  model  parameters   a.   and   b.   are 

i         i 

i  i 

Ex.y.  -  7.  y   X. 
J  J     '  *-~i        J 

1j  --£!_! J^ and         (3.42) 

j-i        j-i 

"St  =  — —  exp  (Y.  +"a\X.)  (3-^3) 

i  iii 

l--a>. 

i 

for  i  =  2,  3,  **»  •••  ,  K.   Note  that  as  with  the  cumulative  failure  rate 
model  the  regression  requires  at  a  minimum  observations  on  the  results 
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of  two  test  phases;  thus,  model  parameter  estimates  are  made  for  the 
second  thru  the  K   test  phase.   The  instantaneous  failure  rate  estimate 

given  by  equation  3-39  was  used  for  the  first  phase  of  testing;  i.e., 

^      2NT      F 


As  the  instantaneous  failure  rate  model  parameter  estimates  were 
obtained,  they  were  applied  to  the  model  in  equation  3-38  to  obtain  model 
determined  estimates  for  the  instantaneous  failure  rate  reliability 
status  at  the  end  of  each  phase  of  testing;  i.e., 

%   =  (l-t^.dT.)  a'  (3.45) 

T.        111 

1 

for  i  =  2,  3,  4,  •••  ,  K.   Again,  the  simulation  index   r   is  left  imp  1 i - 
cit  (r  =  1 ,  2,  3,  • • •  ,  NS IMS  =  100) . 

Upon  completion  of  all  computer  simulations  ( NS I  MS  =  100)  for 
each  specified  lambda  set,  performance  statistics  were  computed  as 


A 


NSIMS 
£  %,         .  (3.*»6) 


T.      NSIMS 

r=l 


"*T.    *' 


NSIMS 


S.D 

1 


1       \~^    />*,       ^Z    ^2 


]£   (^T    "A^  )   ■  and        (3.^7) 


NSIMS-1 

r=l      '>r     ' 


S.D."> 

AT, 


P.S.E.^s.   =   -x  100  (3-^8) 

A. 


T. 

1 


V 


for  i  =  1,  2,  3,  •••  ,  K.   Also,  in  order  to  evaluate  the  forecasting 
performance  of  the  instantaneous  failure  rate  model  the  same  method  of 
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computing  an  estimate  of  the  total  accumulated  test  time  for  a  one  test 

phase  in  the  future  failure  rate  estimate  utilized  for  the  cumulative 

failure  rate  model  in  equation  3-31  was  utilized  for  the  instantaneous 

failure  rate  model  forecasts.   Hence, 

FA     =  (l-a*.Jb\  (TT   )   '   where  (l.kS) 

i  +  l       '   ' 

TT,  +  1   -.TT,  +  (NT.  +  1  xPTT.  +  1)  (3.50) 

for  i  =  2 ,  3 ,  *» ,  •  •  •  ,  K.  Note  again  that  forecasts  were  made  for  test 
phases  3  thru  K  as  was  the  case  with  the  cumulative  failure  rate  model. 
Forecast  statistics  were  then  computed  as 


IMS  IMS 

"t.  =  nsW     E     ^t.     •  (3-51 

r-l 


•^  4 


NSIMS 


^2        (p^T         -   F^T  )      .   and  (3.52) 


, NSIMS-  . 
T.  .  i ,r  i 

i  r=1 


S"D'^T. 


P.S.E./s        =     -xlOO  (3-53) 

FAT. 

i 


FXT. 

I 


for  i  =  3,  k,    5,  ...  ,  K. 
5-   Model  Performance 

a.   Accuracy  Performance 

Figures  3.^+2  thru  3-77  depict  the  accuracy  performance  of 
the  continuous  instantaneous  failure  rate  reliability  growth  model  for 
selected  tests  per  phase  NT.   cases  of  the  lambda  sets  listed  in  tables 
3-1  and  3-2.   In  each*graph  the  specified  underlying  instantaneous 
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failure  rate  X.      from  table  3-1  or  3-2  ( ,  solid  line),  the  mean 

model  determined  instantaneous  failure  rate  X_   from  equation  3-^6 

i 
(0  ,  circles),  and  the  mean  model  forecast  instantaneous  failure  rate 

FX-  from  equation  3-51  (X  ,  crosses)  are   all  plotted  against  the  mean 

i  

total  accumulated  test  time  TT.   from  equation  3-29  for  each  phase  of 

the  specified  reliability  testing  procedure  of  the  acquisition  cycle. 
While  the  specified  underlying  instantaneous  failure  rate  is  plotted  as 
a  smooth,  continuous  line  for  purposes  of  contrast,  it  should  be  noted 
from  both  figures  3-1  and  3-2  and  the  testing  procedure  design  that  the 
specified  underlying  instantaneous  failure  rate  is  actually  a  step  func- 
tion in  shape  with  discontinuities  occurring  at  the  end  of  each  test 
phase. 

Examining  figure  3-^7  for  example  at  a  mean  total  accumulated 
test  time  of  approximately  205-0  test  time  units  the  specified  instanta- 
neous failure  rate  is  0.30,  the  mean  model  determined  instantaneous 
failure  rate  is  approximately  0.2^+,  and  the  mean  model  forecast  instanta- 
neous failure  rate  is  approximately  0.19-   As  noted  on  the  graph  the 
accuracy  performance  plotted  is  for  the  lambda  set  6,  16  test  phase,  20 
tests  per  phase  reliability  testing  procedure  simulation.   Since  the 
point  examined  is  for  the  last  test  phase  (i  =  K  =  16),  table  3-1  shows 
that  for  lambda  set  6,  test  phase  16  the  specified  instantaneous  failure 
rate  is  0.30  as  expected. 

The  instantaneous  failure  rate  model  frequently  yielded  mean 
forecast  failure  rates  that  were  "of f- the-scale"  of  the  accuracy  perfor- 
mance graphs;  i.e.,  mean  forecast  failure  rates  greater  than  1.0.   These 
off-the-scale  mean  estimates  ranged  from  failure  rates  only  slightly 
greater  than  1.0  to  mean  estimated  failure  rates  of  magnitudes  in  excess 
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of  10  .   The  extreme  values  occurred  only  for  the  first  forecast  and 
occasionally  for  the  second  forecast  (test  phases  3  and  k) .   Regardless 
of  the  magnitude  of  these  off- the-scale  forecast  failure  rates,  all 
points  of  magnitude  greater  than  1.0  were  plotted  at  1.0. 

Figures  3-^2  thru  3-^6  and  3-60  thru  3.64  display  the  instan- 
taneous failure  rate  model's  performance  for  the  "nice"  underlying  fail- 
ure rate  progress  paths.   The  model,  while  discerning  the  shape  of 
these  "nice"  underlying  failure  rate  patterns,  is  consistently  optimistic 
in  both  determining  the  current  failure  rate  status  and  forecasting  the 
next  test  phase  failure  rate,  especially  in  the  case  of  the  longer  six- 
teen phase  reliability  testing  procedures.   The  model  performs  more 
accurately  for  the  more  volatile  six  test  phase  procedures  (figures  3-60 
thru  3-6^)  than  for  the  protracted  sixteen  test  phase  procedures  (figures 
3.^2  thru  3-^6).   In  fact  mean  determined  failure  rate  performance  is 
excellent  for  the  shorter  six  test  phase  cases.   Forecasting  accuracy 
typically  is  of f-the-scal e  for  the  first  test  phase  improving  rapidly 
by  the  third  or  fourth  test  phase.   Once  stabilized,  the  mean  forecast 
failure  rate  generally  is  biased  optimistically  as  expected  from  the 
method  utilized  to  produce  forecasts  (equations  3-^9  and  3-50). 

In  figures  3-^7,  3-^8,  and  3-65  thru  3-67  accuracy  of  the 
instantaneous  failure  rate  model  is  displayed  for  essentially  linear 
underlying  failure  rate  progress  paths.   Again,  mean  determined  failure 
rate  performance  shows  that  the  model  detects  the  linear  shape  of  the 
progress  path  with  reasonable  accuracy  but  provides  consistently  opti- 
mistic estimates  of  the  underlying  instantaneous  failure  rate.   Forecast- 
ing accuracy,  while  consistent  in  the  sixteen  test  phase  procedures,  is 
very  poor  being  only  around  50%-60%  of  the  magnitude  of  the  underlying 
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instantaneous  failure  rate  for  the  majority  of  the  test  phases  until 
the  final  one  or  two  phases.   For  the  six  phase  procedures  forecasting 
performance  is  extremely  erratic  as  can  be  seen  in  figures  3-65  thru 
3.67.   In  fact  forecasting  performance  actually  degrades  as  more  test 
information  becomes  available  in  the  case  of  lambda  set  M0D6;  i.e.,  as 
the  number  of  tests  per  phase  increases  from  five  to  twenty  (NT.  =  5 
vs.  NT.  =  20,  figure  3-65  vs.  3 • 66) .   Also,  figure  3-67  is  for  a  twenty 
tests  per  phase  (NT.  =  20)  procedure;  and  therefore,  in  keeping  with  the 
convention  established  for  presenting  accuracy  performance  graphs  this 
figure  shows  the  best  performance  of  the  instantaneous  failure  rate 
model  for  lambda  set  M0D7-   The  five  and  ten  tests  per  phase  performance 
for  lambda  set  M0D7  were  worse  than  the  performance  shown  in  figure  3-67- 

Figures  3-^9  thru  3-52  and  3-69  thru  3-17  portray  the  in- 
stantaneous failure  rate  model's  accuracy  performance  for  permanently 
stagnated  reliability  status  cases.   The  most  vivid  contrast  for  these 
cases  is  between  situations  in  which  failure  rate  stagnates  at  a  high 
value  (A.  =  0.70;  figures  3-^9,  3-68,  and  3-69)  versus  stagnation  at  a 
low  rate  (A.  =  0.05;  figures  3-52  and  3-71).   When  the  underlying  failure 
rate  stagnates  at  a  high  value,  the  instantaneous  model's  mean  determined 
failure  rate  estimates  are   very  erratic  (A.  =  0.70;  figures  3-^9  and 
3.68)  and  the  mean  forecast  estimates  are   virtually  useless  from  a  magni- 
tude consideration  being,  even  once  stabilized,  approximately  40%~50% 
of  the  magnitude  of  the  true  underlying  instantaneous  failure  rates. 

Note  that  figure  3-^9  is  for  a  "best  case"  situation  of  NT.  =  20.   Fore- 

1 

casts  at  best  give  an  indication  of  stagnation  when  the  stagnation  occurs 
at  a  high  true  underlying  value.   In  the  case  of  lambda  set  M0D8  deter- 
mined estimated  failure  rate  improves  significantly  when  more  testing 
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is  performed;  i.e.,  figure  3-69  versus  figure  3-68  (NT.  =  20  vs.  NT.  =  5). 
On  the  other  hand,  when  the  true  underlying  failure  rate  stagnates  at  a 
low  point,  both  the  model's  determined  and  forecast  accuracy  performance 
appear  to  be  excellent  (A.  =  0.05,  figures  3-52  and  3-71,  NT.  =  5) • 
Figures  3-50,  3-51,  and  3-70  show  for  an  intermediate  stagnation  level 
(X.  =  0.30)  how  determined  and  forecast  failure  rate  performance  of  the 
instantaneous  model  transitions  from  poor  to  excellent. 

The  contrast  between  the  underlying  failure  rate  permanent 
stagnation  cases  points  out  a  general  performance  characteristic  of  the 
instantaneous  failure  rate  model.   Because  the  model  produces  consistent 
optimistically  biased  determined  and  forecast  failure  rate  estimates,  as 
the  true  underlying  failure  rate  progress  path  approaches  low  failure 
rates  the  model's  accuracy  performance  improves  significantly  by  neces- 
sity since  the  estimates  become  "sandwiched"  between  the  low  true  under- 
lying failure  rate  and  0.0.   This  "closing  of  the  brackets"  effect  may 
also  produce  favorable  variability  performance  for  the  instantaneous 
fa \ 1 ure  rate  model . 

Figures  3-53,  3-5^,  3-72,  and  3-73  present  the  accuracy  per- 
formance of  the  instantaneous  failure  rate  model  in  the  case  of  true 
reliability  status  progress  interrupted  by  a  period  of  stagnation.   Only 
"best  case"  (NT.  =  20)  graphs  are   shown.   The  model's  mean  determined 
instantaneous  failure  rate  performance  is  fair  for  these  cases  and  con- 
tinues to  exhibit  consistent  optimistic  bias.   Mean  forecast  failure 
rates  generally  are  indicative  of  the  underlying  failure  rate  path  trend 
only  and  are   significantly  off  the  mark  in  forecasting  instantaneous 
failure  rate  magnitude.   The  "kick-down"  in  the  mean  forecast  of  the 
failure  rate  for  the  first  phase  after  the  period  of  stagnation  in  figure 
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3.53  (test  phase  11  at  approximately  130.0  total  accumulated  test  time 
units)  is  more  pronounced  in  the  five  and  ten  tests  per  phase  (NT.  -   5, 
10)  cases  for  lambda  sets  11  and  12. 

For  situations  of  temporary  reliability  status  degradation 
figures  3-55  thru  3-59  and  3-7^  thru  3-77  reveal  that  in  these  "worst 
situation"  cases  the  instantaneous  failure  model  performs  surprisingly 
well  in  discerning  the  shape  of  the  true  underlying  failure  rate  path, 
especially  for  the  twenty  tests  per  phase  procedures  (NT.  =  20;  figures 
3-57,  3-59,  3-75,  and  3-77)-   During  the  period  of  failure  rate  degrada- 
tion, the  instantaneous  model  tends  to  produce  determined  failure  rate 
estimates  that  overstate  the  degree  of  degradation.   This  is  especially 
true  in  the  "least  data"  cases  (NT.  =  5,  figures  3-55  and  3-58)  of  the 
sixteen  phase  testing  procedures.   This  characteristic  is  not  as  pro- 
nounced for  the  six  test  phase  procedures  (NT.  =  5,  figures  3-7^  and 
3-76).   The  model's  mean  forecast  instantaneous  failure  rate  estimates 
remain  optimistic  throughout  the  periods  of  degradation,  but  reflect  the 
shape  of  the  true  underlying  failure  rate  progress  path  with  very  credible 
fidelity.   The  figure  series  3-55  thru  3-57  indicates  how  mean  determined 
failure  rate  estimating  performance  improves  with  additional  testing 
(NT.  =  5,  10,  and  20)  while  mean  forecast  failure  rate  estimating  perfor- 
mance is  very  stable  after  the  third  forecast  for  all  test  per  phase 
sizes.   Again,  note  in  figures  3-59  and  3-77  that  when  the  true  instanta- 
neous failure  rate  has  decreased  to  a  small  magnitude  (test  phase  16), 
the  instantaneous  model's  determined  and  forecast  failure  rate  accuracy 
displays  significant  improvement. 

b.   Variability  (Precision)  Performance 

The  variability  performance  of  the  continuous  instantaneous 
failure  rate  model  is  presented  in  tables  3-15  thru  3-26.   Entries  in 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  2:  16  PHASES,  20  TESTS/pHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  3:  16  PHASES,  5  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  4:  16  PHASES,  5  TESTS/PHASE 
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INSTANTANEOUS    RELIABILITY    GROWTH    M00EL    PERFORMANCE 
LAMBDA    SET    5:     16    PHASES,     5    TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  6:  16  PHASES,  20  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAMB0A  SET  7:  16  PHASES,  10  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMB0A  SET  8:  16  PHASES,  20  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  9:  16  PHASES,  5  TESTS/PHASE 
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FIGURE    3.51 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  9:  16  PHASES,  20  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  10:  16  PHASES,  5  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMB0A  SET  11:  16  PHASES,  20  TESTS/PHASE 
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FIGURE    3.54 


INSTRNTflNEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  12:  IS  PHASES,  20  TESTS/PHASE 
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FIGURE    3.55 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  13:  16  PHASES,  5  TESTS/PHASE 
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FIGURE    3.56 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LPMBDA  SET  13:  16  PHASES,  10  TESTS/PHASE 
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FIGURE    3.57 


INSTflNTfiNEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  13:  16  PHASES,  20  TESTS/PHASE 
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FIGURE    3.58 


INSTANTANEOUS  RELIABILITY  GROWTH  M00EL  PERFORMANCE 
LAMB0A  SET  14:  15  PHASES,  5  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  14;  16  PHASES,  20  TESTS/PHASE 
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FIGURE    3.60 


INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  M001:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.61 

INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D2:  6  PHASES.  5  TESTS/PHASE 
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FIGURE    3.62 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D3:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.63 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  M0DM:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.64 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D5:  6  PHASES,  5  TESTS/PHASE 


127 


o 
o 


o 
o 


-    t    SPECIFIED    INSTANTANEOUS   FAILURE    ROTE 
©     t    MEAN   MOOEL    DETERMINED    INSTANTANEOUS   FAILURE    RATE 
X     t    MEAN   MOOEL    FORECAST    INSTANTANEOUS    FAILURE    RATE 


o 

CD 


<x 
ac 

a 

OCo" 

Z> 


Q 


cr 

ID 

end" 

O 

LU 

cro 

a: 

i— 

(X 

LU 


LUfO 

!Tc5* 


LU 


0 


O 
o 


o 
o 

^.00              3'.  00             6'.00  9 '.00              12.00            15.00 

TOTAL  ACCUMULATED  TEST  TIME 


18.00 


21.00 


FIGURE    3.65 


INSTANTANEOUS  RELIABILITY  GPGWTH  MODEL  PERFORMANCE 
LAMBDA  SET  MODS:  S  PHASES,  5  TESTS/PHASE 
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FIGURE    3.66 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0D6:  6  PHASES,  20  TESTS/PHASE 
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FIGURE    3.67 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  M007:  6  PHASES,  20  TESTS/PHASE 


130 


o 

o 


o 
en 


o 

CO 
LU 

f— 
a 
cc 
o 

CCo" 

_J 

<X 
(0 

o">d" 

O 
LU 

CTo 
I—in 

1° 

1— 
if) 


LU 

£ 

UJcrt  X 

LU 


© 


V 


o 


ol  _  1  SPECIFIED  INSTANTANEOUS  FAILURE  RATE 

O  i  MEAN  MODEL  DETERMINED  INSTANTANEOUS  FAILURE  RATE 
X  t  MEAN  MODEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


o 
a 


^.00  2. 00  il'.OO  6'.  00  81. 00  10.00  12.00        "Tl.00 

TOTAL    ACCUMULATED    TEST    TIME 
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INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  MGD8:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.69 


INSTANTANEOUS  RELIABILITY  GROWTH  MQ0EL  PERFORMANCE 
LAMBDA  SET  M008:  6  PHASES,  20  TESTS/PHASE 


132 


o 

Q 


O 


.  »  SPECIFIED  INSTANTANEOUS  FAILURE  ROTE 
O  »  MEAN  MOOEL  DETERMINED  INSTANTANEOUS  FAILURE  RATE 
X 


.  MEAN  MODEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


o 

CO 


UJ 
\— 

cr 

cc 

o 

CCo" 

ID 


cr 
u_o 

CO 

coo" 

O 
UJ 

z: 

CEO 

i —  u~> 

t— 

CO 

z 


UJ 

X 

Oo 
LUoo 


C_> 
UJ 


o 

X 


0 


o 

o 


o 

o 


^.00  4.00  8.00  12.00  16.00  20.00 

TOTAL  ACCUMULATED  TEST  TIME 


24.00 


28.00 


FIGURE    3.70 


INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMB0A  SET  MQD9:  6  PHASES,  5  TESTS/PHASE 
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INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAM80A  SET  M0010:  6  PHASES.  5  TESTS/PHASE 
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FIGURE    3.72 


INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMB0A  SET  MODI  1 :  6  PHASES,  20  TESTS/PHASE 
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FIGURE    3.73 


INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMB0A  SET  M0012:  6  PHASES,  20  TESTS/PHASE 
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FIGURE    3.74 


INSTRNTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  M0013:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.75 


INSTANTANEOUS  RELIABILITY  GROWTH  MG0EL  PERFORMANCE 
LAMBDA  SET  M0D13:  6  PHASES,  20  TESTS/PHASE 
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FIGURE    3.76 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  M0014:  6  PHASES,  5  TESTS/PHASE 
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FIGURE    3.77 


INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMB0A  SET  M001M:  6  PHASES,  20  TESTS/PHASE 
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these  tables  are  the  percentage  standard  errors  as  computed  in  equations 
3.48  and  3-53-   From  figure  3-47  the  mean  model  determined  instantaneous 
failure  rate  was  read  as  0.24  for  the  sixteenth  test  phase  of  the  lambda 
set  6,  16  test  phase,  20  tests  per  phase  reliability  testing  procedure. 
In  table  3-17  the  percentage  standard  error  corresponding  to  the  sixteenth 
test  phase  of  the  lambda  set  6,  NT.  =  20  simulations  is  24%.   By  equation 
3.48  the  standard  deviation  of  the  instantaneous  failure  rate  model's 
determined  instantaneous  failure  rate  for  this  phase  is  then 


P.S.E.  A    x  X 

S.D.<>   =  !i !i-2^-0.0576  .     (3.55) 

1 16        100         luu 


A  quick  examination  of  tables  3-15  thru  3-26  reveals  that  the 
goal  of  30%  percentage  standard  error  of  failure  rate  estimates  utilized 
in  evaluating  the  variability  performance  of  the  cumulative  failure  rate 
reliability  growth  model  (chapter  III. A. 5. b.)  will  be  too  restrictive  for 
evaluating  the  instantaneous  failure  rate  model's  variability  performance 
Also,  unlike  the  variability  performance  of  the  cumulative  failure  rate 
model  which  almost  uniformly  improved  as  test  phases  of  the  testing  pro- 
cedure were  completed,  variability  performance  of  the  instantaneous  model 
oscillates  for  many  lambda  sets.   For  example,  keeping  in  mind  that  model 
variability  performance  begins  with  the  percentage  standard  errors  re- 
corded for  phase  2,  the  instantaneous  model's  variability  performance  for 
lambda  set  1,  five  tests  per  phase,  determined  failure  rate  in  table  3-15 
improves  steadily  from  76%  in  phase  2  to  }&%    in  phase  11.   In  phase  12 
the  model's  variability  performance  jumps  up  to  41%,  improves  back  to 
32%  by  phase  14  ,    and  finally,  climbs  to  35%  for  the  last  test  phase, 
phase  16. 
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This  oscillation  in  the  variability  performance  of  the  in- 
stantaneous failure  rate  model  means  that  even  if  a  reasonable  percent- 
age standard  error  benchmark  is  selected,  determining  the  first  phase 
for  which  variability  performance  on  all  lambda  sets  is  within  the  goal 
does  not  guarantee  the  performance  goal  will  be  satisfied  for  all  subse- 
quent test  phases  of  the  acquisition  cycle. 

Tables  3-15,  3-16,  and  3-17  display  the  instantaneous  failure 
rate  reliability  growth  model's  variability  performance  for  determined 
instantaneous  failure  rate  estimates  for  the  sixteen  test  phase  proce- 
dures.  If  a  percentage  standard  error  goal  of  h0%    is  set,  the  goal  is 

never  achieved  for  NT.  =  5  or  NT.  =  10  (tables  3-15  and  3-16),  but  is 

i  i 

achieved  for  NT.  =  20  (table  3-17)  on  test  phases  5,  7,    8,  )k,    and  16. 
Again,  the  oscillating  character  of  the  instantaneous  model's  variability 
performance  is  displayed  by  this  method  of  examination.   Tables  3-18, 
3-19,  and  3-20  show  that  in  the  case  of  the  contracted  six  test  phase 
procedures  the  instantaneous  failure  rate  model's  variability  performance 
for  determined  instantaneous  failure  rate  fails  to  attain  the  k0%   goal 
for  all  simulated  tests  per  phase  specifications;  i.e.,  NT.  =  5,  10,  and 
20. 

In  both  the  sixteen  and  six  test  phase  reliability  testing 
procedures  determined  failure  rate  estimates  given  by  the  instantaneous 
model  were  least  accurate  for  lambda  set  8  and  M0D8.   See  figures  3-^9, 
3-68,  and  3-69-   Also,  the  model's  accuracy  performance  for  these  lambda 
sets  was  the  only  case  in  which  no  definite  bias  trend  was  evident.   The 
instantaneous  failure  rate  model's  variability  performance  for  determined 
instantaneous  failure  rate  estimates  is  almost  uniformly  the  worst  for 
lambda  sets  8  and  M0D8  as  evidenced  in  tables  3-15  thru  3-20.   This  was 


U2 


to  be  expected  based  on  the  model's  accuracy  performance  for  these  par- 
ticular specified  underlying  instantaneous  failure  rate  paths. 

Tables  3-21  thru  3-26  contain  variability  performance  results 
of  the  instantaneous  model's  forecast  failure  rates  for  both  the  sixteen 
and  six  test  phase  reliability  testing  procedures.   Percentage  standard 
error  entries  of  1000%  indicate  that  the  actual  percentage  standard 
errors  in  these  cases  were  greater  than  or  equal  to  1000%.   As  with  the 
cumulative  failure  rate  model,  lambda  set  7,  12,  14,  M0D7,  M0D12,  and 
M0D14  provide  difficulty  for  the  instantaneous  model.   These  lambda  sets 
are  the  ones  which  take  the  most  rapid  "plunge"  to  a  low  specified  instan- 
taneous failure  rate  of  \.,    =  0.0500  or  X    ,  =  0.0500.   Again,  if  these 
lambda  sets  are  not  considered,  then  a  k0%   percentage  standard  error  goal 
is  achieved  on  phase  ]k    for  NT.  =  5  and  10  and  on  phase  11  for  NT.  =  20 

I  i 

in  the  case  of  the  sixteen  phase  reliability  testing  procedure  simula- 
tions.  The  instantaneous  model's  variability  performance  for  forecast- 
ing during  six  test  phase  cycles  is  very  poor  achieving  the  k0%   goal  on 
the  last  test  phase  of  the  "most  data"  case  (NT.  =  20)  for  only  seven  of 
the  fourteen  lambda  sets. 

C.   DISCRETE  RELIABILITY  GROWTH  MODEL 
1  .   Model  Description 

The  discrete  reliability  growth  model  evaluated  is  of  the  form 

Rm  =  1  "  exp  (a  +  3m)  (3'56) 

t"  h 

where   R  =  component  reliability  after  the  m   modification  of  the 

m  * 

component 

m  =  number  of  modifications  that  have  been  made  to  the  component, 

and 

a, 6  =  constants  that  determine  the  change  in  the  model  estimated 

component  reliability  as  modifications  to  the  component  are 

accompl i  shed. 
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The  modification  number  m  takes  on  values  0,  1,  2,  .  .  .  with  m  =  0 

designating  the  original  version  of  the  component.   Component  reliability 

R   may  be  thought  of  as  an  instantaneous  or  modification  reliability  as 
m 

opposed  to  a  cumulative  or  average  reliability  since  it  is  the  intrinsic 
reliability  of  the  m   modification  version  of  the  item  rather  than  a 
characteristic  reliability  for  all  m+1  versions  of  the  item  that  have 
been  produced.   Under  the  assumption  of  reliability  growth,  the  quantity 
exp  (a  +  8m)  in  equation  3-56  is  expected  to  increase  as  modifications 
to  the  component  are   made  which  hopefully  improve  its  reliability.   Hence, 
as  modifications  are  accomplished  the  fraction  1/exp  (a  +  (3m)  in  equation 
3-56  decreases  and  component  reliability  increases  toward  the  maximum 
reliability  of  1.0.   The  model  is  considered  discrete  since  the  single 
independent  or  control  variable  m,  modification  number,  is  discrete  as 
opposed  to  a  continuous  variable  such  as  total  accumulated  test  time. 

A  mathematical  analysis  of  the  discrete  reliability  growth  model 
is  given  in  reference  1.   Other  Monte  Carlo  simulation  evaluations  and 
comparisons  of  the  discrete  model  are   contained  in  references  5  and  9- 
2 .   Reliability  Testing  Procedure 

For  the  discrete  reliability  growth  model  the  appropriate  reli- 
ability testing  procedure  of  a  system  acquisition  cycle  is  one  in  which 
modifications  to  an  item  are  made  as  a  specified  number  of  failures  of 
the  item  are  observed  during  the  testing  procedure.   The  modifications 
of  the  item  are  made  to  improve  reliability  and  hopefully  result  in  such 
improvement.   Modifications  are    indexed  by   m  with  a  total  of  NTM  modi- 
fications being  considered  during  an  acquisition  cycle;  i.e.,  m  =  0,  1, 
2,  3,  •••  ,  NTM.   The  total  number  of  item  failures  that  are   observed 
during  testing  of  the  m   modification  version  of  an  item  before  the  item 
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is  modified  aqain  is  denoted  by   NTF   .   The  total  number  of  item  fail- 
J  m 

ures   NTF    is  specified  before  reliability  testing  is  conducted  and  may 
m      r 

be  varied  from  modification  to  modification.  During  testing  of  each 
modification  version  of  an  item,  an  underlying,  inherent  reliability 
that  is  unknown  to  the  project  managers/contractors  is  present  in  the 

t"  h 

item.   The  underlying  modification  reliability  of  the  m   modification 

version  of  the  component  is  denoted  as   R 

r  m 

f  h 

Between  each  failure  of  the  m   modification  version  of  an  item 

a  number  of  tests  of  the  item  are  conducted  which  are  denoted  as   NT   .; 

m,j 

i.e.,  the  number  of  tests  run  on  the  m   modification  version  of  the 

item  between  the  (j-l)    failure  and  the  j    failure.   NT   .   includes 

m,j 

the  j    failure.   So,   NT   .   is  the  number  of  tests  performed  thru  the 

m ,  l 

first  observed  failure  of  the  item  when  the  underlying  modification  reli- 
ability is   R   .   NT  _   is  the  number  of  tests  performed  from  the  first 
m      m,2 

failure  thru  the  second  failure  of  the  item  while  the  underlying  modifi- 
cation reliability  remains   R   and  so  on.   After  NTF    failures  of  the 

m  m 

item  under  test  are  observed,  a  modification  is  accomplished  and  the 

underlying  modification  reliability  of  the  component  changes  to  R   ,  . 

Therefore,  the  failure  number  index  j  runs  from  1  to   NTF   for  each 

m 

modification  m  :  i.e.,  NT   ,  ,  NT   .,  ...  ,  NT  .,__.   .   All   NT   .   are 

m,  1    m,2  m,NTF  m,j 

m 

random  variables  and  cannot  be  specified  beforehand.   This  is  quite  dif- 
ferent from  the  testing  procedure  for  the  continuous  reliability  growth 
models  evaluated  where  the  number  of  tests  per  test  phase  were  specified 
prior  to  any  testing  being  performed. 

Figure  3-78  is  a  schematic  diagram  of  the  discrete  model  reli- 
ability testing  procedure.   The  total  number  of  modifications   NTM   and 

the  total  number  of  failures  permitted  before  modification   NTF   are 

m 
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specified  prior  to  testing.   The  number  of  tests  performed  between  fail- 
ures  NT   .   for  each  modification  version  of  the  item  under  test  consti- 
m,j 

tute  the  data  collected  from  the  reliability  testing  procedure.   As  with 

the  continuous  reliability  growth  models  the  most  salient  feature  of  the 

computer  simulation  of  the  discrete  model  reliability  testing  procedure 

is  that  the  analyst  also  specifies  the  underlying  modification  reliability 

R   that  is  effective  for  each  modification  version  of  the  item.   Thus  an 
m 

accurate  assessment  of  the  accuracy  of  the  model  can  be  made  because  the 

R   values  are   known, 
m 

3 .   Reliability  Testing  Procedure  Computer  Simulation 
a.   Summary 

The  reliability  testing  procedure  appropriate  to  the  discrete 
reliability  growth  model  was  simulated  on  the  Naval  Postgraduate  School 
W.  R.  Church  Computer  Center's  IBM  360/67  System  utilizing  standard  com- 
puter simulation  techniques.   Reliability  testing  procedure  design  para- 
meters  NTM  and   NTF   were  specified  along  with  various  underlying  modi- 
fication reliability  sets.   Tests  of  components  were  then  simulated  with 
component  success  or  failure  being  determined  for  each  test  based  on  the 
appropriate  underlying  modification  reliability   R   .   Number  of  tests 

between  failures   NT   .   data  were  collected  as  each  procedure  simulation 

m,j 

was  accomplished.   Simulations  of  reliability  testing  procedures  for  each 
specified  modification  reliability  set  were  replicated  one-hundred  times 
in  order  to  assess  the  discrete  reliability  growth  model's  variability 
performance. 

After  the  reliability  testing  procedure  simulations  were  com- 
pleted, discrete  reliability  growth  model  estimates  of  the  underlying  modi- 
fication reliability  of  the  component  were  produced  based  on  the  simulated 
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test  data  by  computations  utilizing  ordinary  least  squares  regression 
techniques  to  estimate  the  model  parameters  a  and   3  •   Mean  and 
variability  statistics  of  the  model  reliability  estimates  were  then  com- 
puted to  permit  performance  evaluation  of  the  discrete  reliability 
growth  model . 

b  .   Deta  i 1  Descr i  pt ion 

The  computer  simulation  of  the  reliability  testing  procedure 
appropriate  to  the  discrete  reliability  growth  model  was  initiated  by 
reading  in  the  following  reliability  testing  procedure  design  specifi- 
cations: NTM,  the  total  number  of  modifications  to  be  made  for  the 
acquisition  cycle  under  consideration;  NTF  ,  the  total  number  of  failures 
of  the  m   modification  version  of  the  item  to  be  observed  until  the 

(m+1)    modification  is  made;  R  ,  the  underlying,  intrinsic  modification 

m 

reliability  of  the  m   modification  version  of  the  item;  and  NSIMS,  the 

number  of  times  the  specified  reliability  testing  procedure  was  to  be 

simulated.   For  this  thesis   NSIMS  was  always  specified  as  one-hundred 

s  imul at  ions . 

After  specification  data  were  read  into  the  computer,  uniform 

(0,1)  random  variates  were  drawn  in  sequence  corresponding  to  component 

tests  starting  for  the  modification  0  type  component;  i.e.,  uniform 

random  variates  were  drawn  in  sequence  and  indexed  as   U   .  .   for  m  =  0, 

m,j  ,  i 

1,  2,  ...,  NTM;  j  =  1,  2,  3,  •••  ,  NTF  ;  and  i  =  1,  2,  3,  •••  starting 
with  the  IL  ,  ,   test. 

Each  uniform  variate  was  then  compared  with  the  appropriate 

underlying  modification  reliability   R   to  determine  if  a  success  or  a 

73  m 

failure   occurred    during    the    test.      The   number   of    tests      NT      .      and    the 

m,j 

number  of  failures  accumulated   NF  data  were  recorded  after  each  simu- 

m 

lated  test  such  that 
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NT   .  =  NT   .  +  1  (NT   .  incremented  by  one  test),  and      (3-57) 
m,j     m,j        m,j 

NF   i  f  U   .  .     <  R    (success) 
m     m ,  j  ,  i        m 

NF    =  I  (3-58) 

m      J 

NF  +  1  if  U   .  .  -  R   (fai lure) 
m        m ,  j  ,  1  =  m 


for  m  =  0,  1,  2,  ...  ,  NTM;  j  =  1,  2,  3,  •••  ,  NTFm;  and  i  =  1,  2,  3, 

...  If  the  test  was  a  success,  then  the  simulation  proceeded  to  the  next 

test.   If  the  test  was  a  failure,  then  the  number  of  failures  accumulated 

NF   was  compared  to  the  total  number  of  failures  until  modification 
m         r 

NTF   to  determine  if  a  modification  was  appropriate.   If  NF   was  less 
m  rr   r  m 

than   NTF   ,  testing  continued  for  the  given  modification  and  number  of 
m         3 

tests  were  then  counted  until  the  next  failure;  i.e.,  NT   .  .  .   I f  NF 

m,j  +  l  m 

was  equal  to  NTF   ,  then  a  modification  of  the  item  under  test  was  simu- 
m 

lated  and  testing  continued  for  the  next  modification;  i.e.,  NF   ,   and 
3  m+1 

NT   .  .   being  tallied  next. 
m+ 1  , 1 

Testing  in  this  sequence  was  simulated  until   NF     was  equal 

to  NTFV1_W  which  signalled  the  termination  of  a  given  reliability  test- 
N  I  n 

ing  procedure  computer  simulation.   The  simulation  resulted  in  the  number 

of  tests  until  a  failure  NT   .   data  being  recorded  for  m  =  0,  1,  2,  ...  , 

m,j 

NTM  and  j  =  1,  2,  3,  •••  ,  NTF   .   These  data  constituted  the  total  data 
gathered  from  a  single  computer  simulation  of  the  discrete  model  reli- 
ability testing  procedure  for  a  single  specified  underlying  modification 
reliability  progress  path;  i.e.,  a  specified  set  of  reliabilities  which 
shall  henceforth  be  referred  to  as  a  reliability  set.   The  reliability 
testing  procedure  was  then  replicated  one-hundred  times  (r  =  1,  2,  3,  •••  , 
NSIMS  =  100)  such  that  reliability  testing  procedure  results  data  were 

actually  indexed  as   NT   .    .   Since  ten  reliability  sets  were  utilized 

m,j,r 
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for  evaluating  the  discrete  reliability  growth  model  with  three  different 

total  number  of  failures  until  modification   NTF   specified  for  each 

m 

reliability  set,  3000  reliability  testing  procedures  were  simulated  for 

the  discrete  reliability  growth  model  evaluation. 

k .      Computer  Simulation  Data  Manipulation 

The  discrete  reliability  growth  model  of  equation  3-56  is  proposed 

as  a  model  of  the  unknown,  underlying  modification  reliability   R    in- 

m 

trinsic  to  a  component  as  it  proceeds  thru  a  system  acquisition  cycle. 

It  is  against  this  true  reliability   R   that  the  model's  performance  was 
3  m  r 

measured . 

Equation  3-56  may  be  given  in  the  form 

R  =  1  -  exp  (-a-  6m)  or  (3  -  59/ 

m 

exp  (-a  -  Sm)  =  1  -  R  (3-60) 

m 

for  m  =  0,  1,2,  ...  ,  NTM.   The  logarithmic  transformation  of  equation 
3.60  yields 

(a  +  6m) .  =  -ln(l  -  R  ) .  (3-61) 

J  m  j 

for  m  =  0,  1 ,  2,  . ..  ,  NTM  and  j  =  1 ,  2 ,  3 ,  . . .  ,  NTF   . 

An  unbiased  estimator  of  the  quantity  a  +  6m   is  given  in 
reference  1  as 

0.0  if  NT   .  =  1  (first  test  was  failure) 

m,j 


(a  +  6m)   .  =  {  (3.62) 

m ,  j 

+  —  +  —  +  ...  +  ■ i  f  NT    ^2 

2   3         NT   .  -  1       m,j 

m,j  'J 

for  m  =  0,  1 ,  2,  .. .  ,  NTM  and  j  =  1 ,  2 ,  3 ,  . . .  ,  NTF   . 

m 
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By  equation  3-61 


-  1 n ( 1 -R  Y      .      =   (a  +  6m) 

m  m,j  m,j 


(3-63) 


for  m=0,  1,  2,  ...  ,  NTM   and  j  =  1,  2,  3,  •••  ,  NTF   .   From  this 
relation  an  average  estimate  may  be  computed  as 


NTF 


Letting 


Y  =   -  In ( 1 -R  )   , 

m  mm 


then  equation  3-61  may  be  written  as 


Y  =  a   +   S  m 
mm      m 


(3.65) 


for  m  =  0,  1,2,  ...  ,  NTM  which  is  of  the  form  Y  =  a  +  BX .   Applying 
ordinary  least  squares  regression  estimates  as  given  in  reference  6,  the 


a       and   6   estimates  are 
m        m 


m 


£    (I  -S) 


Y. 


i=0 


m    m 

£    (i  -i) 

i=0 


and 


(3-66) 


a  =  Y  -  S  m 
m        m 


(3.67) 
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for  m  =  1,  2,  3,  •••  ,  NTM  where 


m 
1 


Y  =  -~r  y  Y-   and  (3-68) 

m+1   /  j         i 

i=0 


1 
m  = 


i=0 

Finally,  these  ""a*   and   8   estimates  are  utilized  in  the  discrete  re- 
liability  model,  equation  3-56,  to  produce  the  model  estimates  of  the 
unknown,  underlying  modification  reliability.   These  estimates  are  given 
by 


R  =  1 ! (3-70) 

m 

exp  Co*  '  +  Hh  m) 
m      m 


for  m=  1,  2,  3,  •••  ,    NTM.   Note  than  since  the  regression  procedure  re- 
quires a  minimum  of  two  observations,  model  reliability  estimates  are    pro- 
duced from  the  first  modification  thru  the  last  modification  (NTM).   A 
reliability  estimate  for  original  version  of  the  component  is  derived 
from  equation  3-56  and  3-64  as 

^  -  1 ^^^    •  (3.7D 

exp  (a  +  6m) 

Again,  since  the  reliability  testing  procedure  was  simulated  one- 
hundred  times  (r  =  1,  2,  3,  •••  ,  NSIMS  =  100)  for  each  reliability  set, 
one-hundred  estimates  of  the  modification  reliability  were  obtained  for 
each  modification  version  of  the  component  under  test.   The  mean  value, 
standard  deviation,  and  percentage  standard  error  of  these  estimates 
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were  computed  as 

NS  IMS 


1 


Rm   NS  IMS    Z-rf 

r=l 


Y       "5"  p  ,  (3.72) 

Z— 4  m,  r 


NSIMS 

^•^/n^I  £    (^,r-^)2  'and      ("3) 
mV        r=l 


S.D- 

P.S.E.-^       m x  100  (3-7*0 

m    ^ 
m 


for  m  =  0,  1 ,  2,  ...  ,  NTM. 

To  examine  the  discrete  reliability  growth  model's  forecasting 
capability  a  next  modification  reliability  forecast  was  made  for  modifi- 
cations m  s  2,  3,  ^ ,    •••  ,  NTM.   Forecasting  started  for  the  second 
modification  since  model  parameter  estimates  were  not  available  until 
testing  under  the  first  modification  was  completed.   No  forecast  was  made 
past  modification   NTM   because  there  would  be  no  underlying  modification 
reliability  with  which  to  compare.   Forecasts  were  made  as 


nJ\   =  i ! (3.75) 

m+ 1 

exp  Co*   +  *$     (m+1)) 
m      m 


for  m  =  2,  3,  ^ ,  •••  ,  NTM.   Finally,  forecast  modification  reliability 
statistics  similar  to  the  determined  modification  reliability  statistics 
in  equations  3-72,  3-73,  and  3-7^  were  computed  as 


FR   = 


NSIMS 
1 


Y       FR    ,  (3.76) 

L-j  m,  r 


m   NSIMS  L^i  m,r 

r=l 
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S.D 


NS IMS  

2 


FR   / NS IMS-1 


Y\         (FRm  r  -  FR)   ,   and  (3-77) 


P.S.E.-fJ-  -  -  x  100  (3-78) 

m 


for  m  =  2,  3,  *»,  ...  ,  NTM. 

To  evaluate  the  discrete  reliability  growth  model  ten  reliability 
sets  were  formed  for  use  as  the  specified  underlying  reliability  progress 
paths.   These  reliability  sets  are  listed  in  table  1.21.      All  ten  reli- 
ability progress  paths  start  at  a  relatively  low  reliability  of  0.200 
for  the  original  version  of  the  component  under  scrutiny;  i.e.,  R  =  0.200 
for  reliability  sets  1  thru  10.   The  paths  display  reliability  progress 
that  ranges  from  extremely  rapid  reliability  growth  (reliability  set  1; 

Rn   =  0.200,  R,  =  0.925)  to  linear  growth  (reliability  set  6;  R   ,  =  R  + 
0  I  m+ I     m 

0.150)  to  permanently  stagnated  reliability  progress  (reliability  set  10; 
R   thru  R,.  =  0.200).   Reliability  progress  was  simulated  only  for  compo- 
nents that  were  modified  a  total  of  five  times  during  the  acquisition 
cycle;  i.e.,  NTM  =  5  and  m  =  0,  1,  2,  3,  ^,  5.   Each  reliability  set 
progress  path  was  simulated  for  three  different  total  number  of  failures 

until  modification  specifications;  i.e.,  NTF  =  1,  3,  and  5  failures 

r  m    > 

which  were  held  constant  from  modification  to  modification. 
5.   Model  Performance 

a.   Accuracy  Performance 

Figures  3-79  thru  3-98  present  all  the  cases  (ten  reliability 
sets)  of  the  discrete  reliability  growth  model's  capability  to  determine 
and  forecast  the  unknown,  underlying  modification  reliability  progress 
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path  during  a  reliability  testing  procedure  for  total  number  of  failures 

until  modification  NTF  =  1  and  5-   These  graphs  display  the  "least  data" 

m 

and  "most  data"  cases  which  can  be  contrasted  for  change  and  improvement 

as  the  testing  sample  is  increased.   For  the  NTF  =  3  cases,  the  accu- 

3    r  m 

racy  performance  fell  uniformly  between  the  NTF  =  1  and  NTF  =  5  accu- 

r  '  mm 

racy    performance. 

The  graphs  depict  the  specified  true  underlying  modification 
reliability  R  progress  path  ( ,  solid  line),  the  mean  model  deter- 
mined  modification    reliability      R        from   equation    3-72    for   each   modifica- 

'    m        ^ 

tion  version  of  the  component  under  test  (®  ,  circles),  and  the  mean 


model  forecast  modification  reliability   FR   from  equation  3-76  for  the 
second  thru  the  final  modification  version  of  the  component  (X  ,  crosses) 
plotted  versus  the  modification  number  m  =  0,  1,  2,  3,  **>  5-   Note  that 
the  point  plotted  for  the  mean  model  determined  modification  reliability 
of  the  original  version  (m  =  0)  of  the  component  under  test  is  not  model 
determined;  rather,  it  is  the  mean  value  of  the  estimate  given  by  the 
estimator  of  equation  3-71.   This  point  allows  the  accuracy  of  the  reli- 
ability estimator  utilized  to  be  examined  also. 

To  illustrate  those  quantities  graphed  in  the  accuracy  per- 
formance figures,  observe  on  figure  3-83  that  for  the  third  modification, 
the  specified  underlying  reliability   R.  was  0.90,  the  mean  model  deter- 
mined modification  reliability   R.  was  approximately  0.86,  and  the  mean 
model  forecast  modification  reliability   FR   made  from  the  second  modi- 
fication was  approximately  0.77-   Since  this  graph  is  for  reliability 
set  3,  referring  to  table  3-27,  the  specified  modification  reliability 
for  the  third  modification  of  reliability  set  3  is  0.900  as  is  graphed 
on  figure  3-83-   Finally,  note  on  figure  3-83  for  the  unmodified  version 
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of  the  component  tested  (m  =  0)  that  the  reliability  estimator  of  equation 

3. 71  produced  a  mean  estimate   Rn  of  approximately  0.175  for  the  true 

underlying  reliability   R   of  0.200. 

o 

With  the  exception  of  reliability  set  9  (figures  3-95  and 
3.96)  the  discrete  model  determines  the  shape  of  the  underlying  reliability 
progress  path  with  very  good  accuracy.   Forecasting  accuracy,  while  fairly 
good  for  the  "nice"  reliability  progress  path  sets  1  thru  *t,  displays 
difficulty  with  the  anomalous  reliability  progress  paths  characterized 
in  reliability  sets  6  thru  10.   The  discrete  model  occasionally  produced 
negative  mean  forecast  estimates  of  reliability.   The  negative  mean  fore- 
cast reliability  estimates,  the  magnitude  of  which  never  exceeded  0.20, 
are   plotted  as  0.0  on  the  accuracy  performance  graphs.   For  example  see 
figure  3-97  for  mean  forecasts  at  modifications  2,  3,  and  ^4.   No  negative 
mean  forecasts  of  reliability  were  produced  when  the  total  number  of 
failures  until  modification  was  specified  as  five  (NTF  =  5);  i.e.,  the 
"most  data"  case. 

The  discrete  model  accuracy  performance  graphs  display  the 
marked  improvement  that  takes  place  both  in  determining  and  forecasting 
reliability  status  as  the  specified  total  number  of  failures  until  modi- 
fication  NTF    is  increased  from  one  to  five  which  essentially  increases 
m  ' 

the  test  data  which  the  model  can  utilize.  The  improvement  in  accuracy 
is  also  displayed  in  the  NTF  =  3  accuracy  performance  graphs  which  are 
not  presented.  The  reliability  estimator  of  equation  3-71  displays  the 
same  improvement  in  accuracy  when  the  testing  sample  size  is  increased 

as  may  be  seen  by  examining  the  graph  pairs  for  NTF  =  1  and  NTF  =  5 

m  m 

at  the  modification  m  =  0  point. 

Figures  3-79  thru  3-98  reveal  that  the  discrete  reliability 
model's  accuracy  performance  for  determining  the  underlying  reliability 
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is  in  general  slightly  pessimistic  in  the  instances  when  mean  estimates 
are  not  "on  the  money"  as  in  figures  3-84  and  3-86.  Except  for  the  re- 
liability sets  1  and  2,  NTF  =  5  cases  shown  in  figures  3-80  and  3-82, 

m 

mean  model  forecasting  accuracy  tends  to  run  from  slightly  pessimistic 

as  in  figure  3-90;  reliability  set  6,  NTF  =  5  to  grossly  pessimistic  as 

in  figure  3-91;  reliability  set  7,  NTF  =  1  or  figure  3-97;  reliability 

set  10,  NTF  =  1.   The  general  trend  evidenced  by  the  graphs  is  that 

the  less  test  data  available  (NTF  =  1)  the  more  pessimistic  the  mean 

m 

model  estimates,  determined  or  forecast.  The  same  is  true  of  the  reli- 
ability estimator  of  equation  3-71- 

Reliability  set  9  which  characterizes  a  reliability  growth 
pattern  interrupted  by  a  period  of  reliability  degradation  is  the  only 
set  with  which  the  discrete  model  experiences  a  significant  problem. 
While  not  detecting  the  full  magnitude  of  the  degradation,  the  model's 
mean  determined  reliability  estimates  do  display  the  period  of  degrada- 
tion credibly.   See  figures  3-95  and  3-96.   However,  the  model's  mean 
forecast  modification  reliability  estimates  are  quite  contrary  to  the 
true  underlying  reliability  progress  path.   The  significant  problem 
being  that  the  period  of  degradation  is  not  forecast  until  the  true  under- 
lying reliability  status  has  recovered  from  the  period  of  degradation  and 
is  in  fact  displaying  significant  reliability  growth.   This  behavior  is 
apparent  even  for  the  "most  data"  case  NTF  =  5  in  figure  3. 96.   While 
model  forecasting  capability  suffers  in  the  "least  data"  case  for  reli- 
ability set  10,  NTF  =  1,  the  discrete  model  copes  with  permanent  reli- 
ability stagnation  satisfactorily  as  seen  in  figures  3-97  and  3-98. 
b.   Variability  (Precision)  Performance 

Tables  3-28  thru  3-33  present  the  discrete  reliability  growth 
model's  variability  performance  for  determining  and  forecasting 
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the  reliability  status  of  a  component  as  it  undergoes  modification  in  a 
system  acquisition  cycle.   Entries  in  the  tables  are  percentage  standard 
errors  as  computed  in  equations  3-7^  and  3.78.   For  example  from  figure 
3.83  the  mean  model  determined  reliability  for  modification  3  from  reli- 
ability set  3  was  found  to  be   R_  =  0.86  in  the  NTF  =  1  case.   From 

table  3-28  the  percentage  standard  error  P.S.E/C^  corresponding  to  this 

R3 
mean  model  determined  modification  reliability  is  20%.   Therefore,  by 

equation  3-7^  the  standard  deviation  for  the  observed  mean  model  deter- 
mined rel iab  i 1 i  ty  is 

P.S.E.^ 
S-°-T3  =  V^00    =0.86x^=0.172  .  (3.79) 

In.  tables  3-28,  3-29,  and  3-30  the  discrete  model's  vari- 
ability performance  for  determining  modification  reliability  status  is 
displayed  for  the  specified  total  number  of  failures  until  modification 
cases  of  one,  three,  and  five  failures,  respectively.   Two  salient 
characteristics  displayed  by  the  results  are  (1)  the  large  variability 
of  the  estimates  produced  by  the  reliability  estimator  of  equation  3-71 
and  (2)  the  large  variability  of  the  reliability  estimates  produced  by 
the  discrete  model  for  reliability  set  10,  the  permanently  stagnated 
reliability  case.   The  same  difficulty  is  displayed  to  a  lesser  degree 
for  rel iab  i 1 i  ty  set  7 • 

If  reliability  set  10  is  not  considered,  then  a  goal  of  kQ% 
percentage  standard  error  is  achieved  by  the  discrete  reliability  growth 

model  on  the  fifth  modification  for  the  NTF  =  1  and  3  cases  and  on  the 

m 

fourth  modification  for  the  NTF  =  5  cases.   If  reliability  set  7  is 

m 

also  dropped  from  consideration,  the  k0%   goal  is  achieved  on  the  fifth 
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modification  for  NTF  =  1 ,  on  the  third  modification  for  NTF  =  3,  and 

m  m 

on  the  second  modification  for  NTF  =  5-   Note  that  the  reliability 

m 

sets  characterizing  temporary  reliability  stagnation  and  degradation 

(sets  8  and  9)  did  not  have  to  be  taken  out  of  consideration  in  order  to 

find  variability  performance  at  a  reasonable  level. 

Tables  3-31,  3-32,  and  3-33  present  the  discrete  model's 

variability  performance  in  forecasting  the  reliability  of  the  subsequent 

modification  version  of  the  component  under  test  for  the  NTF  =1,3,  and 

m 

5  cases.   Again,  the  worst  variability  performance  occurs  for  reliability 

sets  7  and  10.   Dropping  reliability  sets  7  and  10  from  consideration,  a 

h0%   percentage  standard  error  goal  is  achieved  only  in  the  case  of 

NTF  =  5  at  the  fourth  modification.   Hence,  variability  performance  of 
m 

the  discrete  reliability  growth  model  for  forecasting  is  not  particularly 
good. 
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IV.   CONTINUOUS  FAILURE  RATE  RELIABILITY 
GROWTH  MODELS  COMPARISON 

Codier  in  reference  2  gives  a  derivation  in  which  an  equation  of  the 
form  of  the  continuous  instantaneous  failure  rate  reliability  growth 
model  (equation  3 • 37)  is  obtained  by  differentiating  the  equation  of  the 
continuous  cumulative  failure  rate  reliability  growth  model  (equation 
3.1)  with  respect  to  the  total  accumulated  test  time   TT  after  substitu- 
tion of  the  cumulative  failure  rate  estimator  of  equation  3.13'   The 
derivation  essentially  proceeds  along  the  following  line: 

XTT  =  ==■  =  3TT    ;   therefore,  (O) 

TF  =  BTT(1"a)  .  (k.2) 

Now, 

!rr--3TT  (STT°"a))  -  0-a)8TT"a  ,  and  (4.3) 

Hence  Codier  derives  an  equation 

XT  =  (l-a)3TT"a  (4.5) 


and   claims    that    the   equation    is    the   continuous    instantaneous    failure 
rate  model 


■a 


XT     =    (1-a)    bTTd    ;    i.e.,  {h. 6) 
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equation  3-37.   The  implication  being  that  the  cumulative  model  parameters 
a  and   3  are  equivalent  to  the  instantaneous  model  parameters   a  and 
b  ,  respectively;  and  so,  once  either  a  and   3  or  a   and  b  have 
been  estimated,  it  is  unnecessary  to  estimate  the  other  parameter  pair. 

The  cumulative  and  instantaneous  failure  rate  models  were  considered 
as  two  unrelated  models  which  were  proposed  to  model  different  aspects  of 
reliability  growth  from  the  viewpoint  of  this  thesis.   The  evaluation  of 
the  two  models  was  conducted  accordingly.   In  order  to  investigate  the 
proposition  that  the  cumulative  and  instantaneous  models  are  related  thru 
equivalence  of  model  parameters  a  hybrid  cumulative  failure  rate  model  was 
devised  that  utilized  the  parameter  estimates  generated  for  the  instanta- 
neous failure  rate  model.   Also,  a  hybrid  instantaneous  failure  rate 
model  was  devised  that  utilized  the  parameter  estimates  generated  for 
the  cumulative  failure  rate  model.   Specifically,  the  hybrid  cumulative 
failure  rate  model  is  given  by 

X*     =  "S\   TT.     I'r  (4.7) 

TT .       i , r   i , r 
i  ,r 

where  a.    and  b.    were  determined  by  the  instantaneous  failure  rate 
i  ,r        i  ,r 

model  equations  3.42  and  3 - ^* 3 -   The  hybrid  instantaneous  failure  rate 
model  is  given  by 

^Xl  =  (1  -1x\   )  T.        TT.    1>r  (4.8) 

T.  i  ,  r    i , r    i ,r 

i ,r  '      '     ' 

where  'a*.    and   3.    were  determined  by  the  cumulative  failure  rate 
i,r        i  ,r 

model  equations  3.21  and  3-22  after  appropriate  transformations  given 
i  n  chapter  I  I  I .A. 4. 
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As  cumulative  and  instantaneous  failure  rate  model  parameter  esti- 
mates were  determined  for  each  phase  of  each  simulation  for  the  differ- 
ent lambda  sets  they  were  applied  to  the  hybrid  models  as  well  as  to 
their  appropriate  models.   Hybrid  model  estimates  of  the  cumulative  and 
instantaneous  failure  rates  were  recorded  and  the  standard  statistics 
of  chapter  I  I  I  were  compi led. 

Table  4.1  contains  the  mean  cumulative  and  instantaneous  determined 
failure  rate  estimates  produced  by  the  hybrid  models  for  the  lambda  set 
3,  ten  tests  per  phase  (NT.  =  10)  case  listed  with  the  appropriate  under- 
lying mean  test  time  weighted  average  cumulative  failure  rates   ATT 

i 
(equation  3.30)  and  the  specified  underlying  instantaneous  failure  rates 

A.   (table  3.1).   Lambda  set  3  is  one  of  the  "nice"  failure  rate  sets 
for  which  both  the  cumulative  and  instantaneous  failure  rate  models  dis- 
played some  of  their  best  accuracy  and  variability  performance  (see 
figures  3.5,  3.6,  3.7,  and  3.44;  and  tables  3-4  and  3.16).   Entries  of 
10.0000  for  the  hybrid  cumulative  model  mean  determined  cumulative  failure 
rate  in  table  4.1  indicate  that  the  actual  mean  estimate  was  greater  than 
or  equal  to  10.0000. 

The  performance  of  the  hybrid  models  displayed  in  table  4.1  for  a 
"nice"  lambda  set  is  typical  of  the  behavior  of  the  hybrid  models  for 
all  the  lambda  sets,  nice  or  anomalous.   The  hybrid  cumulative  model 
(instantaneous  model's  estimated  parameters  substituted  into  the  cumula- 
tive model)  produced  failure  rate  estimates  of  magnitude  greater  than 
10.0  for  the  first  few  test  phases  followed  by  very  erratic  estimates 
(including  negative  mean  estimates)  which  displayed  no  relation  to  the 
mean  test  time  weighted  average  cumulative  failure  rates  supposedly  being 
modelled.   On  the  other  hand  the  hybrid  instantaneous  model  (cumulative 
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TABLE  4.1 

HYBRID  CUMULATIVE  AND  INSTANTANEOUS  FAILURE  RATE  MODEL  SIMULATION  RESULTS 
LAMBDA  SET  3:   16  PHASES,  10  TESTS/PHASE 


MEAN  TEST 

HYBRID 

HYBRID 

TIME 

CUMULATIVE 

INSTANTANEOUS 

WEIGHTED 

MODEL  MEAN 

MODEL  MEAN 

AVERAGE 

DETERMINED 

SPECIFIED 

DETERMINED 

CUMULATIVE 

CUMULATIVE 

INSTANTANEOUS 

INSTANTANEOUS 

FAILURE 

FAILURE 

FAILURE 

FAILURE 

PHASE 

JL  -U  JL  JL  JL 

RATE 

RATE 

RATE 

RATE 

1 

0.7000 

0.6702 

0.7000 

0.6702 

2 

0.4703 

10.0000 

0.3530 

0.3605 

3 

0.3565 

10.0000 

0.2380 

0.3048 

4 

0.2863 

10.0000 

0.1800 

0.2669 

5 

0.2392 

10.0000 

0.1450 

0.2362 

6 

0.2064 

-0.1003 

0.1220 

0.2173 

7 

0.1818 

1.9321 

0.1060 

0.2016 

3 

0.1625 

-0.0171 

0.0933 

0.1889 

9 

0.1474 

-0.0065 

0.0837 

0.1796 

10 

0.1346 

0.0532 

0.0760 

0.1684 

11 

0.1241 

-0.0206 

0.0697 

0.1626 

12 

0.1152 

0.0237 

0.0644 

0.1548 

13 

0.1076 

0.0070 

0.0600 

0.1497 

14 

0.1009 

0.2491 

0.0562 

0.1442 

15 

0.0952 

0.0518 

0.0529 

0.1402 

16 

0.0900 

0.0060 

0.0500 

0.1357 
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model's  estimated  parameters  substituted  into  the  instantaneous  model) 
produced  what  appeared  to  be  a  good  failure  rate  estimate  for  the  first 
one  or  two  phases  (i  =  2,  3;  models  do  not  estimate  for  the  first  phase) 
but  quickly  transitioned  to  a  dampened,  sedate  pattern  of  estimates  which 
was  very  unresponsei ve  to  even  the  most  radical  variations  in  the  under- 
lying instantaneous  failure  rate  progress  path.   Also,  the  hybrid  in- 
stantaneous model's  failure  rate  estimates  after  the  second  or  third 
phase  generally  exhibited  increasing  magnitude  error  except  for  crossing 
situations  in  the  cases  of  temporary  stagnation  or  degradation  of  the 
specified  underlying  instantaneous  failure  rates. 

Regarding  the  negative  failure  rate  estimates  produced  by  the  hybrid 
cumulative  model  (instantaneous  model  parameters  in  cumulative  model), 

the  following  observations  on  the  parameter  estimates   a!"    and  b. 

y  H  i  ,r        i  ,r 

of  the  continuous  instantaneous  failure  rate  model  were  made  during  the 
reliability  testing  procedure  computer  simulations.   During  the  simula- 
tions the  magnitude  of  the  parameter  estimate   a.    was  continually  very 
close  to  1.0  often  being  more  or  less  than  1.0  only  in  the  tenth, 
eleventh,  or  twelfth  decimal  place.   From  the  instantaneous  model  equa- 
tion k.G    it  can  be  seen  that  logically  if  the  a.    estimate  is  less 

i,r 


than  1.0,  the  b.    estimate  should  be  positive;  and,  if  the  a.    est 
i  ,  r  K  '  i  ,  r 


i  - 


mate  is  greater  than  1.0,  the  b.    estimate  should  be  negative  in  order 
3  '       i  ,  r  3 

to  produce  positive  estimates  of  the  instantaneous  failure  rate.   Exactly 

such  behavior  was  observed  during  the  computer  simulations;  as  the 

magnitude  of  the  a?  estimate  moved  to  either  side  of  1.0  the  sign 

i  ,  r 

of  the  b.    estimate  would  "flip-flop"  accordingly.   Of  course  if  a 
negative  b.    estimate  is  plugged  into  the  cumulative  failure  rate  model 
for   6   in  equation  k.] ,    a  negative  estimate  of  the  cumulative  failure 
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rate  necessarily  results  no  matter  what  the  sign  or  magnitude  of  the 

"a/    estimate  which  is  substituted  for  a   (save  a  value  of'^>   =  °°)  . 
i ,r  i,r 

Hence,  it  is  very  plausible  for  the  hybrid  cumulative  model  to  have  pro- 
duced negative  estimates  of  cumulative  failure  rate. 

In  contrast  to  the  hybrid  cumulative  model's  erratic  failure  rate 
estimates  the  hybrid  instantaneous  model  (cumulative  model  parameters 
in  instantaneous  model)  produced  failure  rate  estimates  which,  after  one 
or  two  test  phases,  displayed  a  "sluggish"  growth  pattern  and  generally 
increased  in  magnitude  error  from  the  underlying  instantaneous  failure 
rate  A.   being  modelled.   This  situation  is  not  surprising  because  the 
cumulative  model  parameters  being  employed  in  the  hybrid  instantaneous 
model  are  parameters  utilized  in  modelling  a  "smoothed"  quantity,  the 
underlying  cumulative  (average)  failure  rate  that  is  collectively  charac- 
teristic of  all  the  versions  of  an  item  tested  thru  a  given  point  in  the 
acquisition  cycle.   As  the  history  of  the  item  stretches  farther  and 
farther  (total  accumulated  test  time  TT  increases)  the  cumulative 
failure  rate  of  the  item  becomes  increasingly  insensitive  to  the  current 
instantaneous  failure  rate  intrinsic  to  the  item.   Hence,  the  parameters 
of  the  cumulative  model  likewise  become  increasingly  insensitive  to  the 
current  instantaneous  reliability  status.   When  these  relatively  stable 
parameters  are   utilized  in  another  model  whatever  its  form,  "smoothed" 
stable  output  is  reasonably  expected. 
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V.   EVALUATION  SUMMARY  AND  CONCLUSIONS 

This  thesis  examined  three  reliability  growth  models  for  which  accu- 
racy, preci s ion,  and  robustness  characteristics  were  investigated  over  a 
wide  variety  of  true  underlying  growth  patterns,  both  "nice"  and  anomalous 
The  evaluation  of  a  continuous  cumulative  failure  rate  model,  an  instanta- 
neous failure  rate  model,  and  discrete  reliability  model  was  accomplished 
by  computer  simulation  of  reliability  testing  procedures  that  were  appro- 
priate to  the  reliability  growth  model  types  for  a  systems  acquisition 
cycle.   Performance  of  the  models  both  in  tracking  and  predicting  true 
underlying  reliability  progress  patterns  was  measured  for  mean  accuracy 
and  variability  (precision)  over  progress  patterns  which  included  most 
situations  that  can  be  encountered,  both  good  and  bad.   The  models'  per- 
formance characteristics  are  summarized  in  the  following  paragraphs. 

A.   CONTINUOUS  CUMULATIVE  FAILURE  RATE  RELIABILITY  GROWTH  MODEL 

The  continuous  cumulative  failure  rate  model  examined  generally  dis- 
played very  good  to  excellent  accuracy  performance  in  tracking  both  "nice" 
and  anomalous  true  underlying  cumulative  failure  rate  patterns.   The 
cumulative  model  did  exhibit  difficulty  in  determining  and  forecasting 
true  underlying  patterns  that  characterize  rapid  reliability  growth 
(failure  rate  decay)  in  the  latter  phases  of  the  systems  acquisition 
cycle.   The  difficulty  was  evidenced  by  mean  model  determined/forecast 
cumulative  failure  rate  estimates  that  diverged  from  the  true  underlying 
values  on  the  pessimistic  side;  thus,  the  cumulative  model  provided  esti- 
mates that  could  be  considered  as  upper  bounds  on  failure  rate  in  this 
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situation.   The  other  anomalous  situation  with  which  the  cumulative  model 
experienced  difficulty  in  tracking  and  predicting  accuracy  was  one  of 
reliability  growth  interrupted  by  a  period  of  reliability  degradation. 
The  modelled  cumulative  failure  rate  path  both  lagged  the  true  underlying 
cumulative  failure  rate  path  and  failed  to  reflect  the  full  magnitude  of 
the  failure  rate  degradation.   Thus  dependence  on  the  model  in  this  situ- 
ation could  hamper  response  to  the  degradation  by  giving  a  "too  little 
too  late"  signal  of  the  problem.   The  cumulative  model  coped  with  true 
underlying  cumulative  failure  rate  patterns  that  were  permanently  stag- 
nated quite  adequately. 

As  the  reliability  testing  procedures  of  the  acquisition  cycles  com- 
pleted more  test  phases,  the  cumulative  model's  variability  (precision) 
performance  improved  un i formly  for  both  "nice"  and  anomalous  true  under- 
lying failure  rate  paths.   Variability  percentage  standard  error  goals 
were  satisfied  earlier  and  earlier  in  the  acquisition  cycle  as  more 
testing  data  was  made  available  to  the  model.   This  good  variability 
performance  provides  a  degree  of  confidence  in  employing  the  cumulative 
failure  rate  model  in  actual  systems  acquisition  programs  where  the 
intrinsic  cumulative  failure  rate  status  is  truly  unknown. 

B.   CONTINUOUS  INSTANTANEOUS  RAILURE  RATE  RELIABILITY  GROWTH  MODEL 

The  continuous  instantaneous  failure  rate  model  examined,  while 
exhibiting  the  capability  to  track  and  predict  the  shape  of  various 
unknown,  underlying  instantaneous  failure  rate  path  types  with  very  good 
accuracy  and  robustness,  consistently  produced  optimistically  biased 
estimates  of  the  instantaneous  failure  rate.   If  bias  must  be  accepted 
in  a  reliability  growth  model,  then  pessimistic  bias  is  preferred  to 
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optimistic  bias.   Consistent  pessimistic  bias  permits  managers  to  utilize 
the  model  produced  estimates  as  lower  bound  type  values  and  have  confi- 
dence that  reliability  status  is  being  observed  from  the  "right  side  of 
the  fence".   On  the  other  hand  consistently  optimistic  model  produced 
estimates  impart  a  great  deal  of  uncertainty  as  to  how  bad  reliability 
status  might  be  and  what  degree  of  corrective  action  is  required. 

The  anomalous  situations  in  which  the  instantaneous  model  experienced 
difficulty  in  tracking  and  predicting  the  shape  of  the  true  failure  rate 
path  were  (l)  the  initial  period  of  recovery  in  reliability  growth  after 
a  period  of  temporary  reliability  stagnation,  (2)  permanently  stagnated 
reliability  growth  at  high  failure  rates,  and  (3)  reliability  growth  in- 
terrupted by  a  temporary  period  of  reliability  degradation.   In  the  case 
of  temporary  reliability  degradation  the  instantaneous  model  charted  a 
determined  failure  rate  path  that  was  an  exaggeration  of  the  underlying 
instantaneous  failure  rate  path;  i.e.,  the  periods  of  growth  were  displayed 
optimistically  and  the  period  of  degradation  was  portrayed  to  a  magnitude 
greater  than  it  was  in  truth.   The  instantaneous  model's  forecasts,  while 
accurately  capturing  the  shape  of  the  underlying  progress  path,  were  con- 
sistently optimistic  during  the  period  of  degradation. 

The  instantaneous  model's  accuracy  performance  was  generally  better 
for  contracted  acquisition  cycles  and  suffered  when  the  reliability  test- 
ing procedure  was  extended  to  encompass  a  large  number  of  test  phases. 
Because  of  the  model's  consistent  optimistic  bias,  accuracy  performance 
improved  as  the  true  underlying  failure  rate  decreased  thereby  "sandwich- 
ing" the  model  estimates  between  the  true  failure  rate  and  0.0.   Also, 
the  instantaneous  model  routinely  furnished  "of f-the-sca le"  forecasts  for 
the  first  one  or  two  phases  of  an  acquisition  cycle. 

The  instantaneous  model's  demonstrated  capability  to  determine  and 
forecast  the  true  underlying  reliability  status  was  overshadowed  by  its 
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poor  variability  performance  which  (1)  never  achieved  a  really  comfort- 
able percentage  standard  error  goal  uniformly  for  all  failure  rate  pat- 
terns simulated  and,  more  damaging,  (2)  oscillated  as  test  phases  of  the 
reliability  testing  procedure  were  accomplished.   Because  of  this  poor 
variability  performance,  the  instantaneous  failure  rate  model  cannot  be 
employed  by  itself  with  any  degree  of  confidence. 

C.   DISCRETE  RELIABILITY  GROWTH  MODEL 

The  discrete  reliability  growth  model  examined  generally  displayed 
very  good  determined  reliability  accuracy  performance  even  for  the  most 
restricted  test  data  cases.   Slight  pessimistic  bias  was  demonstrated  for 
model  determined  reliability  status  while  a  greater  magnitude  of  pessi- 
mistic bias  was  present  in  model  forecast  reliability.   Again,  pessimis- 
tic bias  is  favored  over  optimistic  bias  in  charting  a  reliability  prog- 
ress path.   The  degree  of  pessimism  in  determined  and  forecast  reliability 
decreased  markedly  as  more  test  data  on  each  modification  version  of  an 
item  were  gathered.   In  cases  of  limited  test  data  the  discrete  model 
often  forecast  negative  estimates  of  reliability  for  the  early  modifi- 
cation versions  of  a  component  under  test. 

For  the  anomalous  situation  of  temporary  reliability  degradation  the 
discrete  model  (1)  failed  to  determine  the  full  magnitude  of  the  degrada- 
tion and  (2)  provided  reliability  status  predictions  that  lagged  the 
true  underlying  reliability  path  outcomes  significantly.   In  actual  test- 
ing this  performance  characteristic  would  give  a  delayed  signal  of  a 
situation  that  required  corrective  action.  Also,  gathering  additional 
test  data  failed  to  remedy  this  deficiency  in  the  discrete  model's  per- 
formance. 
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Variability  performance  of  the  discrete  model,  while  generally  good, 
revealed  difficulty  with  the  permanently  stagnated  reliability  case  and 
the  case  of  rapidly  increasing  reliability  during  the  latter  modifica- 
tion versions  of  an  item.   Variability  performance  improved  uniformly  for 
all  progress  path  characterizations  as  more  modifications  of  the  item  be- 
ing tested  were  accomplished.   Also,  variability  goals  were  satisfied 
earlier  in  the  acquisition  cycle  as  more  complete  testing  was  accomplished 
on  each  modification  version  of  the  item.   This  nice  behavior  of  the  dis- 
crete model  lends  confidence  to  its  utilization;  and  therefore,  the  dis- 
crete model  is  preferred  to  the  continuous  instantaneous  failure  rate 
model  for  obtaining  a  measure  of  the  current  or  "instantaneous"  reli- 
ability status  of  an  item  proceeding  thru  an  acquisition  cycle. 

D.   GENERAL  OBSERVATIONS 

For  determining  current  reliability  status  it  may  be  suggested  that 
rather  than  employing  the  reliability  growth  models,  why  not  utilize  the 
point  estimators  of  reliability  status  which  were  used  to  provide  data 
to  the  models?  Although  the  performance  of  the  point  estimators  appro- 
priate to  each  reliability  growth  model  were  only  observed  at  one  point 
for  each  model  in  the  reliability  testing  procedure  simulations  (first 
phase  of  testing  or  mod  0  version  of  a  component),   at  that  point  the 
estimators  displayed  very  poor  variability  performance  upon  which  the 
reliability  growth  models  improved  rapidly  and  significantly  which  tends 
to  negate  any  confidence  in  utilizing  the  point  estimators  based  on  their 
good  accuracy  performance. 

In  retrospect  the  performance  characteristics  demonstrated  by  the 
reliability  growth  models  examined  suggest  that  as  a  model  is  employed 
the  estimated  reliability  progress  path  it  produces  be  compared  with  the 
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appropriate  model  accuracy  performance  graphs  in  chapter  III.   If  the 
estimated  reliability  progress  path  corresponds  to  a  case  for  which  the 
computer  simulation  performance  results  show  the  model's  performance 
was  good,  then  confidence  can  be  placed  in  the  estimated  progress  path. 
On  the  other  hand  if  the  estimated  path  corresponds  to  one  of  the 
anomalous  cases  where  simulation  results  revealed  a  deficiency  in  the 
model's  performance,  then  the  simulation  performance  results  at  least 
give  an  indication  of  which  direction  the  true  underlying  reliability 
progress  path  lies. 

Finally,  although  use  of  the  continuous  instantaneous  failure  rate 
reliability  growth  model  is  questionable,  since  the  test  data  required 
for  the  instantaneous  model  is  identical  to  the  data  collected  for  the 
continuous  cumulative  failure  rate  model,  application  of  the  instanta- 
neous model  simultaneously  with  the  cumulative  model  utilizing  the  method 
described  in  the  preceding  paragraph  may  provide  some  insight  to  the 
shape  of  the  true  underlying  instantaneous  failure  progress  path  from 
the  instantaneous  model.   But  the  comparison  performed  between  the  two 
continuous  failure  rate  models  definitely  indicates  that  the  cumulative 
failure  rate  and  instantaneous  failure  rate  models'  parameters  not  be 
interchanged  based  on  their  hypothetical  equivalence. 
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APPENDIX  A 


Derivation  of  Ordinary  Least  Squares  Regression  Estimates  of  the 
Continuous  Instantaneous  Failure  Rate  Reliability  Growth  Model  Parameters 


The  continuous  instantaneous  failure  rate  model  reliability  growth 
model  as  given  in  equation  3-37  may  be  written  as 

X.    =  b(l-a)TT."a  .  (A.l) 

Taking    the    logarithmic   transformation   of    this    equation   yields 


In    X.    =    In   b(l-a)    -    a    In   TT.    .  (A. 2) 


Let 


Therefore, 


2>E 

i  =  l 


Y.    =    In    X.       , 
i  i 


X.    =    In   TT.     , 
i  i 


-     2>, 

Y      = ,      and 


X      =^- 


Y.    =    In    b(l-a)    -    aX.  (A. 3) 
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The  residual  error  for  regression  is 

E.  =  Y.  -*?.  =  Y.  -  [In  b(l-a)  -  aX.]  .  (A.*0 

The  goal    of    the    regression    is    to  minimize    the   sum   of    the    residual    errors 
squared;    i.e., 

min     ^Tej2   =      V]  (Y  .    "  ^j )  2   =      ]T  [Y  .  -  In   b  ( 1  -a)    +  aX  .  ]  2. 

(A. 5) 
Taking    partial    derivatives    of  equation   A. 5  with    respect    to      b      and      a    ; 
then    equating    the   partials    to   0   yields    the   following    two   equations    in 
two   unknowns: 

(1)  3    ]T£.2 

-gg =     £[2    (Y.    -In  b(l-a)    +   aX.)(-  ^y  (1-.)]        -  0 

(2)  3    £e.2 

-j; £[2  (Y;  "ln  b(l"a)  +aXi)(-  brnr(-b)+xi)]  -° 

Simplification  of  (1)  and  (2)  yields  in  sequence: 

>      [Y.    -    In   b(l-a)    +   aX.]    =   0    ,    and 

(2') 

^T[(Yi    "ln   b(l-a)    +   aX.)     (/jla)    +   X.)]    =   0    .      Then, 

c,,,)     ZYi =  2>  b°-a)  ■  EaXi  ' 

(2") 

£TFi7Yi  +    EYixi  =  ElT^T1nb(1-a)  +    2>i   lnb(1"a)  " 

aX 


JljT^T'     ZaXi2  '     FinallY> 
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0'") 


y^Y.    =   n    In   b(l-a)    -a    /*!  X.    ,      and 


(2,M)    "n^y    2>i  +    EYixi  =xtV  1n  b(1"a)  +  (ln  b(1'a))     Sxi  ' 


TPiT 


Ds  - a  £  x; 


Equation    (1MI)    may   be  manipulated   such    that 

(1"")       In   b(l-a)    =  -     TV    +  -     Vx.    =   Y   +   aX      01 
(r,,M)      b(l-a)    =   exp    (Y   +   aX)     . 

Finally,  the  estimate  for  b  is 


b  = 


77^7 


exp  (Y  +  'a*  X)  . 


(A. 6) 


Substituting  equation  A. 6  into  (2MI)  yields 


^"")  TFiy  X>-  £v.-Tnrl"[iafeJji  ('-a)]  + 


,     rexp    (Y   +   aX)     n       n      v^v  a  \^v  v-"\    2 

ln[     rri) — (1"a)j   Lxi  -  TTaj  2^x;  -  a   l/i 


Clearing  the  logarithms  in  (2"")  and  transposing  leaves 


(2",M)  -rArr   (Y  +  aX)  +  (Y  +  aX)   Vx 


TFaT 


Z; 


mr 


!>•  - a  E 


(T^F 


£*•-  z: 


Y.  X 


Expanding  (2 )  yields 


(2*) 


-7^-T-    X    +     Y 


TFI7  T      TT17 


Exi+ax    Exi  -Ti^r  Z; 


ZXi      "  TNaT     X/i    =     Z 


Y.    X. 
i       i 
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Substituting  and  clearing  terms  in  (2*)  leaves 


(2**)         -TT-^-T 


T^aT 


E 


TTaT 


Exi +  ^  Zxi  +  ax  E 


TPTT 


E*,- 


EXi2-TTaT    £Yi    =      EY|X1    • 


Solving    for      a      in    (2**)    gives 


(2-0      a(X    2x;    -     £x.2)   =    E 


Y.X.    -    Y     Y^  X.      or 
i     i  ^— '      i 


a     = 


2>ixi  "  9    Ex; 


£*,-   2>i 


(A. 7) 
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